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ABSTRACT 

This report describes the developsent of a design and 
a saall working model for a system for teaching and assisting in the 
performance of bibliographic on-line searches through the use of an 
intermediary computer* The system in its present form is intehded for 
direct users of scientific and technical information system^s**- 
Bovever, it is felt that the intermediary computer may become th6 
standard way to receive information about, gain access to, and make 
productive use of the increasing number of complex and non standard 
computer services* Appended are tutorial programs and a transcript of 
a test use of the model* (Author/STS) ^ 



* Documents acquired by ERIC include many informal unpublished * 

* materials not available from other source^* ERIC makes every effort * 

* to obtc^in the best copy available* Nevertheless, items of marginal. 4i 

* reproducibility . are often encountered and this affects the quality 

* of the microfiche and hardcopy reproductions ERIC makes available * 

* via the ERIC Document Reproduction Service (EDRS) « EDRS is not * ♦ 

* responsible for the quality of the original document* Boproductions ♦ 

* supplied by EDRS are the best that can be made from the original* ^ * 



ERIC 



Lrvl 

-5M 



Individualized Instruction for Data Access (IIDA) 
Final Design Report 



us DEPARTMENT OF HEALTH, 
EDUCATION & WELFARE 
NATIONAL INSTITUTE OF 
EDUCATION 

HIS docOment has been repro. 

UCED EXACTLY AS RECEIVED FROM 
ME PERSON OR ORGANIZATION ORIGIN- 
TING IT POINTS OF VIEW OR OPINIONS 
TATED DO NOT NECESSARitY REPRE- 
E^^T OFFICIAL NATIONAL INSTITUTE'OF 
OO^ATlON POSITION OR POL»CY 



Prepared by 



Charles !♦ Meadow, Principal Investigator 
Thx>mas !♦ Hewett 
D* Jean Raf snider 
David E. Toliver 

Drex^l University 

Philadelphia, Pennsylvania 19104 



Bernard Epstein ^ 
Janet V* Edelmann 
Ann Maher 

The Franklin Institute 

Research Laboratories 
Philadelphia, Pennsylvania 19103 



ERIC 



Under Grant number DSI-76-09737 from the Division of Science Information, 
National Science Foundation 



/ 



Published by the Graduate School of Library Science, Drexel University, 
Philadelphia, Pennsylvania 19104. July, 1977 ♦ - * 



O 



ABSTRACT 



. Individualized instruction for Data Access (IIBA) is a project whose 
intent is to provide active assistance to users of on-line bibliograpllic data 
base searching^ systems while they are solving an information search p.rdblem. 
It also provides instruction prior to use* The system is being pfCmarily 
developed for scientists and engineers who are the end-users of scientific and 
^technical information. Assistance is provided through an intermediary 
computer which monitors the user^s interaction. with a data base processor, 
analyzes his performance and provides assistance to him in the conduct of 
the search. Tutoring, in the fotm of. computer-assisted instruction, is also 
provided through the same computer. 

The diagnostic procedures were developed in recognition that there 
is no accepted' mode.l of a successful search, but that there are certain i<jiown ^ 
errors or practices which can be machine-detected and whose occurrence will 
deter successful results. In using IIDA, a user is searching a pub^licly 
aiyailable data base through a commercial search service. IIDA is interposed 
between him and the data base' computer. 

, Evaluation and planning for implementation ind productive use- of the 
system are included in this report. 



KEYWORDS: data base search, information retrieval, search strategy, 
lian-machine communication, user performance, user performance evaluation, 
^earch intermediaries, computer-assisted instruction', information system 
fisers, information systems evaluation, information systems marketing. 
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PREFACE 



This report covers the design of a system for Individualized 
Instruction in Data Access (IIDA) , a method for teaching and assisting, 
in the performance of bibliographic on-line searches entirely through 
use of an intermediary computer ♦ The purpose of the project is to 
develop a method of teaching potential users of scientific and technical 
bibliographic data bases how to use these resources* Preliminary tests 
indicate the method can be successful and the design indicates that its 
cost should not add materially po the cost of direct use of the data 
base search services, and may reduce it. 

Bibliographic searching is a problem-solving activity. The com-*^ 
plete plan for performing a search cannot be specified in advance. ' 
Learning the mechanics of searching — -the commands one .may 'issue to the 
computer — does not constitute learning how to search. There is tech- 
, nique or strategy to 'be learned as we^l. Because of the ill-defined 
nature of searching, thei:e can be no mechanistic way to measure the 
correctness of a particular users search. IIDA is a heuristic system ^ 
it depends upon techniques that appear to 'work* even if not soundly 
based in theory. ^ \ 

IIDA is ^ not solely a teaching system. Its most important role is 
to assist^ in the performance pf searches, helping users in the actual 
performanc^ of a rea^L task. 

The concept' of IIDA is new, but we envision it as becoming the 
routine way, in the future, for* users to deal with the increasing number 
of complex and non-standarized computer services. The intermediary 
computer will become the standard way to receive instruction about, gain 
access to, and make productive use of these many on-line facilities. 

IIDA^is intended, in its present form, ♦primarily for direct users 
of scientific and technical information ^systems. These *are typically 
scientists and engineers who perform a small ntamber of bibliographic 
searches per year and who have not found working through a technical 
library to have been a productive process in the past. 

The system is not intended to supplant .the professional inter- 
mediaries, but to enable those who do not use library services to gain 
access to information systems. We believe that greater understanding by 
users of how these systems operate will also enhance" the ability of 
users to work successfully with intermediariers* 

This, repofet concludes the first year, of work on^the IlDA project. 
We undertook to design an operational IIDA system, to produce a working 
model, and to develop plans for use and evaluation of the technology. 

In Section 1 of the report we discuss the objectives and the over- 
all plan for development of IIDA. 

Section 2 is a descripticm of the instructional subsystem. It is a 
performance-oriented description of IIDA; a set of performance require- 
ments. 

Section 3 is a technical description of the computer system, hard- - 
ware and software, needed to implement the requirements set forth in 
Section 2. 
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Section 4 discusses plans for implementation of the system and its 
initial use, marketing and evaluation. : 

Section 5 contains a description of the working model of IIDA and 
the early results of its use. 

Appendix A contains specifics of proposed course outlines. 

Appendix B contains the transcript of an actual use of IIDA during 
our testing period* " 

For any reader desiring a quick overview of the system, Section 1 
might suffice. Readers not concerned with computer and programming 
details mJ.ght prefer to skip, at least initially, Sectidn 3. 

IIDA has been a joint project of the Graduate School of Library 
Science, Drexsl University, and the Franklin Institute Research Labora- 
tories. The project has been sponsored by the Division of Science 
Information, Naticaal Science Foundation. We acknowledge ^/ith ..thanks 
the guidance given us by Dr. Joel GoldKar and Dr. Carole Ganz of NSF. 
We also wish to thank Dr. Robert Rich of Princeton University, who* 
worked with us and gave invaluable advice as a consultant on evaluation 
and utilization planning. 
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1. • OBJECTIVES,Ain) OVERALL PLAN OF DEVELOPMENT 



1.1 The IIDA Concept ^ 

IIDA - Individualized Ihstruction for Data Access is a system 
that provides instruction in the performance of a man-comput;er , inter- 
active activity. IIDA can administer conventional computer- assisted 
instruction, a mode in whicl^ IIDA takes the initiative, provides in- 
structiopal'^material to a studetit, and , asks questions about. it. Its 
uniqueness, however, comes from monitoring the interaction between 
student and computer, evaluating the performance of the student, and 
communicating to him information about errors detected and positive 
sugge^ions for proceeding. In this mode, IIDA assists a user in the, 
performance of a complex task. ^The interactive activity t>"eing taught is 
bibliographic searching. It is the intent of .jtfhe project to produce a 
system which can teach science students or scientists how to perform 
searches without ; requiring them to take classrScm instruction. 

Mechanically, IIDA m^kes use of an Intermediary computer which is 
placed between the student; user and a data base processor. The inter-*- 
mediary computer administers computer-assisted instruction and performs 
analyses of student actions, maintains student records and provides tfte 
conversational programs that are Vised» to hel^ the student underst^and his 
errors and his options. The IIDA intermediary computer does not itself 
search a bibliographic data base. Instead, it communicates with a^ 
commercial database search service so that the student, even whiie 
learning, is working with a teal data base and a real search*^ service. 

IIDA operates in any of three instructional modes: tutorial, * 
exercise, and assistance. In tutorial mode, IIDA is a computer-assisted 
instruction system. In this igode, students learn search mechanics, data 
base structure, search strategy an'd how to use the special IIDA features 
not' available from the data base service. , * 

In exercise* mode, students may work on searches of their own 
choosing, but are constrained in how they perform the search. IIDA 
expects them ^to try all commands available and to search in a manner ' 
prescribed by' the system. The purpose of the 'exercise mode is to give 
students .practical experience in use of the commands -r it is not to 
accomplish a searching objective. ^ - 

In assistance mode, the student works on a search of his own 
choosing and' his objective is taj accomplish that search successfully; 
IIDA does not constrain him in choice of commands or their sequencing. 
IIDA does provide him a great deal of ' diagnostic assistance if he needs 
It, the approximate equivalent of a coach helping him to do the search, 
but not telling }iim, step by step, how to do it. * 

In the exercise and assistance modes, IIDA does not prestore « 
answers to questions and expect the students to match these answers.^ 
The system cannot 'know' how to solve the student's problem. The 
diagnostic facilities are the heart of the system. They heip^ the, 
student understand what errors he has made or why he has failed fco 



progress in his search. They offer ^ternative possxjilities. The 
better the student's performance, theVess of IIDA's diagnostics he will 
<5ee*. For users who make only occassional bibliographic searches and who 
are likely to forget the details of theVommand language from time 
to time, IIDA can always be used. That rs, IIDA is not a system which 
provides a few hours of instruction apd^ then leaves students on their 
own. They can always use IIDA in assistance mode to help them solve 
an actual working .problem." 

In addition to administering the computer-assisted instruction and 
the diagnostics, the IIDA computer performs a number of , user- invisible 
data communications- tasks. One of these is to serve as a concentrator, 
so^ that several users can share a single communication line between the , 
IIDA computer and the data base processor. This^line can operate at a 
higher speed than most user termi^ials and can hold an incoming message 
until some free time is available on the jline. The cost of the IIDA 
computer will be far less than that of the data base processor; hence 
the cost of ^xising the data base, processor through IIDA and its con- 
centrator need be no more than the cost of going direct, with a, single 
line per user. After the diagnostics — this is the second* major in- 
novation of IIDA — it can reduce the cost of using data base services,' 
even for e:^ei:ienced users. 

The grimary intended users .of IIDA are the scientists and engineers 
who are the end-users of the information. Normally, users work through 
an intermediary who .may be a librarian or a specialist in^ the user'^ 
discipline. We are departing from what has become tradition in 'biblio- 
graphic searching. But, there has been no proof of the eff4.cacy of the 
intermediary system and little discussion of alternatives. IIDA is an 
alternative . In Section 1.2.5 we discuss some recent studies of infor- 
^mation use involving intermediaries. 

Offering -a service not involving intermediaries does not mean that 
we propose to abolish them.* As end-users are trained and use IIDA for ' 
occasional searches, they can become, more effective partner's when they 
do work with intermediaries on the-^more difficult search problems. 

3 » ' * 

1.2 Characteristics of the User Group 

We are concerned with five categories of user characteristics: 
some personal characteristics, such as/educational background an^ 
cpmputer experience; how^'they perceive a search to be organized*; how 
they select commands arid make use of them; their behavior as a market 
for information systems; and user attitudes toward humafi intermediaries. 

'1.2.1 Personal characteristics , 

The stude;cit is assumed to be a scientifically trained persoii. 
Vhile this has" no rigorous definition and it' is not important for us to 
have one, we mean by it^an engineer, chemist, physicist, biologist, 
psychologist, etc., who, has or is pursuing the equivalent of at least a 
J)achelor*s degree in some field of -science. Thus, even if not familiar 
with the literature of his field, the student will know that there is a 
^literature, will know the vocabulary of his field and should not be 
botheted by the task of learning a n^w'skill which is la^rgely^ mechan- 
icar/ Indeed, a high proportion of students will have studied computer 
programming or have at least worked with programmers or computer output. 
We should not have to contend with the problem of overcoming a student's 
resistance to the idea of using a computer in a "humanistic" endeavor. 
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Similarly, although the student may not be formally trained in 
mathematics, the notion of se't or of, the elementary boolean set opera- j 
tions should be either already known or triv:^ il to learn. At any rate, 
these concepts should not be a major intellectual challen<3e, either to 
student or teacher. 

'A key assu'C tipn about the' student is that ne is a serious^ pro- 
fessional person who wants to learn this new sk^ll. It is not a re- 
quirement of IIDA tutorial material, therefore, to create in the student 
the basic motivation^^o learn bibliographic searching. However, since 
we haye also assumed that the student is a person who, for what.ever 
reason, has elected not to work through an intermediary in his searching 
efforts, we^must avoid jgiving the impression that this is a trivial 
task, easily mastered in an hour or two of instruction. In particular, 
we would like to be able to motivate him ^at least to be willing to ■ 
acquire a tlser^s manual to use as a reference and as a means of self-- 
instruction J.n learning other data bases or\ search systems than ve will 
teach. ^ 

Some students will be motivated to learn more -.about on-line search- 
ing and become more frequent, direct users of the existing sea*:ch ser^ 
vice's. Others will elect to continue their future wor1c through IIDA. 
Still others, having gotten an initial exposure, may wish in the future 
to work with an intermediary. When this last group uses librarian 
services, their new knowledge will enable them to work in a completely 
new relationship with librarians than previously. They will know the' ' 
vocabulary and much of the mechanics of searching. Without frequent, 
practice, they^^wTTl not know the current contents of the data base and. 
this will prevent them from becoming expert searchers. Because of their 
^new"* ability to communicate with the intermediary in the latter^s tejrms, 
they may well become far mor^ effective users of information systems 
than ever before. ^ ^ 

1.2;2 Gegeral organization of a search 

Penniman [Iwl] identifies four phases of a search: Index search, 
logic fonnulation, document dis^)lay (actually record display - the docu- 
ments are raicely represented iri bibliographic retrieval systems) , and 
other (including off-line print and such administrative commands as 
BEGIN, END or FILE, the last of which changes the file being searched). 
In Lockheed DIALOG terms, index search would i?hclude the EXPAND command 
ani^ SELECT when it ^as usred with a single descriptoi;. When SELECT is 
used with more than one descriptor for full text searching, we would 
place it in the logic formation ca'tegory. 

Standera -[1.2] categorizes commands in a slightly different way: 
start/stop, executive (all logic and* display commands), supportive 
(index search, set history, etc.)> informative (information about the 
data base or proce4ures for" search)'^ and auxiliary (e. g. ordering copies 
of hard copy documents) . 

For the purposes of. our tutorial approach, we have used Penniman's 
phases modified somewhat by Standera* s: 

r. ^ndex search. (Search of thesaurus or inverted files on a 
single term.) ^ 

2. Logic formation. (Combinations of sets or text search using a 
combination of descriptors.) 

3. Record display. ^ 

4. Procedural. (These are commands that do npt produce search* , 
results but are necessary to organize the search, e.g. BEGIN, END, 
FILE.) , ■ , 

9^- Jl 'J \ 




5. Diagnostic, (These are requests for infgnnation or for as- 
sistance in recognizing errors or**poor performance •) , 

The* first three of these categories define thq.-6"rder of p^roceeding 
that we will teach our students, that is that ^a search begins with 
commands of the first type anci proceeds through the second to^he third. 
Procedural commands and diagnostics come where they' must • This order' of 
searching seems to have* some degree of najturalness to users and so will 
be used during the early stages of training. Students will not be re-, 
quired to^ follow this order in doing their own searches. 

1*2,3 Selection of commands and timing 

Penniman and Standera both repibrt on approximate timing of searches 
and, very roughly, they indicate that a typical search has these charac- 
teristics: ^ ' • , . 
^ ^ Duration:. 15 - 30 tainutes _ 

Niomber o'f command's usedt about 15 

Interval between commands: 1-2 minutes 

Pennim^ s data on elapsed time indicate a negative exponential, curve, 
with mode in the range <j>f 0-5 minutes per search, and nfeap about 15 



minutes* 



We lack' good data on how long .it tates C, \LOG to execufe a command, ^ 
but our intuitive f eelii^gs^ and unrecorded experience suggest that it" 
..takes far Itess tharf 1 - '2 minutes consumed by users. > ThCis, most of the 
time the user*s t^erminallis either^free or being used for output typing, 
but since the communications lines are full duplex, there is considerable 
• unused capacity iri the entire ' system, 

The research reports \we have used' do not describe users precisely. 
We assume that they were relatively experienced searchexs;, ^We further ? 
assume that less experienced seaTrchers will. consume more Lime between 
Commands^ but not necessarily use more total.elapseld time. They may ffeel 
pressured to minimize their total time. ^ > I 

Standera states that only a relative handful of different commands 
% are actually us'^d in.most\ searche's. Users, therefore, are generally 

\ confining tH^mselves io a small subset of the total language. This 

behavior also 'follows our institution-and information observation in 
* D^fexel*s Ihfonnation Systems Laboratory. It suggests to us that the ^ 
teaching of the commands be broken into segments: first the basic 
^ command set, then practice with its use, then the advanced command sep 

and -then i^ractice with Its use*. The advanced training may be made 
optional. ^ 

1.2.4 Market behavior ' " 

• In developing the concept of IIDA, it had been our original intent- 
to take -it directly to industrial scientists for use — this, after all, 
was the user group for whom it was primarily developed. More inves^ti- 
gation 3.ed to the conclusion that this would an 'unusually dif ficult 
grpup to introduce to an unproyen system. W6 decided, therefore, to 
' begin testing IIDA with college engineering stude: and graduate stu^ 
tf dents. Industrial scientists or engineers will also be 'tested for their 

« reactions and for comparison with stu'^ent reactions. ^The main point is 
. -to be able to 'sell' people on the xxsk of IIDA. It is a new service 
for which a laarket must be created and the market is national or even 
n international in scope. By working,, even for several years, primarily 
wifh student groups,, a user demand can .be created as' graduates move into 
positions in^ industry and government. \ * 

ERIC ' ■ • \ 
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Most of the market studies we i have found relate to the use of data 
base search services through the mediation bf a library. Wish and Wish v 
[1.3] 'for example, report on their own .work and survey that of others In 
this domain. Because IIDA i% primarily directed coward delivering data ^ 
base services directly to t^w4 user "v^tl out^^an intermediary and to offer 
him the possibility of continuing to work through. IIDA even after training 
is complete, we feel these studies are not particularly relevant to our ^ 
project. ^ 

Some 'work will r>e done in the next phase of IIDA development on 
planning market strategies. The concentration, aowever, will be on ' 
proving the ^fficacy of the techriiqvies. Only by actual experience wit.h 
some users can we begin to make ^spjecifie, plans for marketing. Henipe.,. 
the question of acceptability to industrial users is being postponed,' 
not dropped from consideration. • " ' ^ . 

1.2.5 , The setting for on-line services and'^'the use of Intermediaries 

The way the on-line searching' indust^ has evolved, provision of 
its! services through a library, with a librarian a^ an intermediary, is 
the norm. Probably the 'principal exception is that in some highly 
technical organizations, typically firms or laboratories in the chemical 
industry (including petroch'^uiical ,and -pharmeceiitical) . In- these a 
termknal may be located org-'nizationaily closer to end-users* and the 
intermediary is likely to have a professional degree and perhaps a titje 
sucH ^as informatidn chemist . - , • 

Little if any consideration has been given to studying whether this 
state of affairs is best or what the alternatives might be.' In partic- 
ular, we find no ^ research on end-use.r reaction to the quality of inter- 
mediary services although there is evidence that library shunuers con- • 
stitute a sub&::antial portion of any non-li'brary population [1.4]. 

Thus, we' feel justified in assuming t|;iat IIDA is, indeed^ aiming 
for a previously unexplored market — a market for search services pro;- 
vided directly to the end user, in his own organizational environment,^ 
without the necessity of an intermediary, but also witj^out- denying the 
possibility of using one should he so elect. Hence, rather than bas*e^ 
our marketing on previously done work, we must carefully distinguish 
between user s^tudies basied on *in-library use and olir own projected^ in- ^ 
office use. ^ 

Two of the best ^nown. studies of ^user behavior .ware those of the*^ 
SysteiAs Development Corporation — Impact of On-*Line Retrieval ^Services 
[1«5] --.and Lockheed* s Investigation of the Public >Library as a Linking , 
Agent to^Maj or .Scientific, Educational, S>^cial and Environmental Data 
Bases [1.6']. These studies were concerned "lorimarily (SDC) or exclu- 
sively (Lockheed) with in-libr,ary use of systems. » Both implicitly 
assumed .the use of a professional intermediary and neither evaluated the ' 
^ end-user *s attitudes toward the quality of the intermediary's, services* 

1. The SDC -Study . SDC notes that search services aimed primarily 
at "individuals in organizational units other than the library coramunitfy" ' 
such as Mead Data Corporation and the N*ew York Times Information Bank 
were^ excluded from the study^p. 2). * • 

A figure possibly significant for IIDA is that among "infrequent, 
searchers" slightly more than half (56%) felt "fairly comfortable and . 
efficient" at the terminal and the remainder "barely comfortable and not 
very efficient" or "had to releam the system every time I use it." 
About 22% of the sample surveyed put themselves in the infrequent 
searcher category (again, recall, they are almost always librarians or 
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their managers, not end-users). All judgments of searcher efficiency 
aye. self-made (p, 67). ^ 

' Some data is provided on search time: separate, figures for 
search times when the respondent was learning and "now," as the survey 
was conducted. Both distributions are so flat as to suggest that there- 
are other, unreported variables involved;, that is, t.he .probability of a 
searcii taking t^ minutes, untirt, gets fairly high (over 94 for begin- 
ners) ^is nearly equal for all .t. Penniman's data [1.1] on the other 
hand, shows a -markedly-different distribution, but is based on a dif- " 
f erent user population/'<|\v 68) . . 
: We concur with the 'following: 

Our study shows that most on-line bibliogr'nh-^c searches 
are performed 'by information intermediaries ^ . ^ early days of 
on-line- systems, there was a strong belief ^^^..^ many designers' and 
planners that these systems should be designed for, and used by, 
, endndsers. This belief -is still held by . some but it has not been 
trarislated into prkctice. However, on-line systems permit — and, 
some would say, demand a hew relationship between the end-users 
and the information intermediary, and the challenge today ij to 
define the kinds of interfaces that takt full advantage of the 
potential of the on-line, interactive technology, (p.- 193) 

This confirms the lack of study of the intermediary process for on- 
line systems. 

Our conclusion- from the SDC study is that it hints at the existence 
^of a body of .users for an IIDA-lifce system but does not directly address 
the, problem. 

2. The Lockheed Study . This project is, of course, intended to be 
concerned with a library setting. In the stud^ there was' a period 
during which users had free access to' DIALOG followed by. a period during 
which half the usual rate was charged. Average Si^arch time ranged from 
about 30 to 18 to 15 minutes as the cost and user experience changed. 
These figures are comparable to jPetiniman's. 

Abourhalf the*users, high. for' average users of a puT)lic iibrarv, ^ 
were scientists of some kind (p. 3-5)..* This is likely. then, to have' 
been a sample of the eventual IIDA user group. 

During the period when users had to pay for service, 80% said the 
' service would be \iseful several times a year, "^several" being undefined . 
but consistent with the IIDA assumption (p. 3-7). - 
^ • Lockheed cites a problem (p*. 4-4) of maintaining ifamiliarity with 
"all data bases [as] one 'of the* most difficult faced by ref'-zence 
librarians conducting the o: -line searches." This is a hiv\t of a 
problem with intermediary services and suggests the possible value of 
IIDA to- tlje 'end user or the intermedimy. ^ ^. 

1.3 Objectives of the*Project' 

X 

' The general purpose of the Individualized Instruction* for Data 
Access project is to teach users of b'ibliogf aphic interactive infor- 
mation retrieval systems how to search and to "assist them in the per^- 
formance of searches. It is aimed primarily a^t the end-^user of infor- 
mation, rather than- the professional intermediWy, or librarian-^ The 
typical prospective user was characterized in the previous sectionT^ 
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It is .imporjtaht to stress that the primary target group is one 
whose members are ri&t--now bibliographic searchers to any important 
extent.' Thus, IIDA does not attempt to lure tliem away from other forms 
of bibliographic system usage; it attempts to provide yet another option 
for those who wish to become users* 

Why the concentration on users, rather than intermediaries? First, 
a negative answer. It is not because. of ^any lack of interest in or 
appreciation for the<,role of interrfiediary. More positively, it is 
because this prospective tiser group, the true consumers, is a large and . 
often unserved one. ^We do not forsee any dimunition of the overall role 
of intetmediaries, however successful IIDA may be. On the contrary, we 
see an increase. We believe there is a valid analogy with the compu- 
tational use of computers. Prior to the development of FORTRAN in 1957, 
programming was almost exclusively in machine or assembly language (a 
language one-f or-one with machine language nmmands) . Such programming 
took much time to learn avd was tedious to carry out. One result was 
that virtually a^l programming was done by prof essional programmers - 
the intermediaries of their day. Often, the problem of communication 
between user and programmer surpassed that of the technical design and 
production of the program. The user could not speak in programming 
terms. The programmer did not always understand the user's vocabulary 
and needs. Result: dei^ay, .high cost, frustration. We suggest it is 
often so today with bibliographic searching The intermediaries do not 
always speak the user's language (there are, of course, exceptions). 
The user has not the knowledge to. speak in search system vocabulary. 
Result: poor communication, poor results, frustration. 

PORTRAIT and subsequent high-order programming languages made it 
possible for people with some technical backgrounds to become program- 
mers. Often, they coujLd write their own small programs faster and more 
easily than^ could a professional, when interpersonal communication 
time was considered. Bid this replace the professional programmer? 
Indeed not. The demand for professional programmers is as vigorous 
today as 20 yeard ago/ But professional programmers more often work on ^ 
large, more complex projects, the ones that the partly- ski lied user 
cannot handle. lurthermore, the partly-skilled ^user can now talk to the 
professional programmer in the latter' s language, vastly reducing the 
communications problems between them. The user is a better. -user of 
intermediary services than when he was entirely untrained in programming . 

So it will be with bibliographic ^searching, we feel. When users 
have some personal knowledge of the content and mechanii;s of on-line 
systems, they will be better able to communicate and use the services 'of 
an intermediary. When the 'job to be done^is simple, they can do it 
themselves. * ' ' 

The formal objectives of this prdject are: 

1. To teach searching . This will include tutoring in data 
base structure, 3earch strategy, search mechanics , and the use of 
the.UDA system and it will include providing^ assistance to users 
'in the performance of actual searches. 

2. To demonstrate the validity of the IIDA approach .' This 
will be done both by evaluating the system in use under test conditions 
andTby demonstrating the extent of users of the system, once completed. 

3. To improve the use of scientific information services * 
The successs of IIDA depends not Only on making the system work 
mechanically, but pn bringing bibliographic information services to ' . 
more people. * , • - • 



To make th e, system self-supporting , IIDA is a new idea in 
training and it is^ experimental in nature. Its success will tie 
beneficial to the science and technology ^community and eventuailly'' 
.to other fields. It is therefore a proper project for support by 
the National Science Foundation. It is an objective, however, 
eventually to make IIDA not only self-supporting, b;ut to bring it 
to the point at;' whieh it can support additional res'^earch in the 
field. , • 

5* To extend the technique to, other applications . Not a 
direct part of this project, but- a long term objective of pro^eet ^ 
members is the extension of this training method to othfer fields. 
Presently, we feel it will become applicable and useful in teaching 
the use of any complex but structured operation done through at 
computer, such as the use of packaged computer programs. 



1.4 Project Development Plan ^ . . 

To accomplish the objectives set forth above will require a develop 
ment effprt lasting several years. The major phases are: (1) Design, 

(2) Implementation, (3) Initial Use, (4) Revision, (5) Extended Use and 

(6) Evaluation. • . , 

The purpose of the design phase was to produce technical specif i- 
c^ations for the system and a plan for implementing them. Hiis report . 
presents the results of that phase and brings it to a conclusion. 

The implementation phase will produce the working computer system, 
programs; courseware, and evaluative techniques needed to bring the 
complete system to real users. It is the longest and most expensive 
phase. Its analogous task in the physical sicence world is construction 
of the apparatus . upon which to ^perform experiments. To justify the 
assumption, that IIDA can actually be successfully implemented, a working 
model- was constructed during the design phase. This "is reported in ' 
Section 5. '* ^ 

, Following implementation will be a period of closely monitored 
initial use and formative evaluation. In this phase,, actual users who 
will be engineering student'^' at Drexel will use the system in conjunc- 
tion with routine ciassrpom assignments. It will be important during ? 
this phase to make others aware of the system 'and its.' accomplishments to 
prepare the way for extended use^, in Phase 5. Th'is. can , come abqut 
through workshops on the system, demonstrations and papers in profes-. 
sional journals. Details of this planning will be developed during ' 
Phases 2, Implementation, and 3, Initial Use. 

The fourth phase is called Revision. Based upon feedback from the 
initial -user group,', c^ianges will almost certainly be required before the 
system can be put Into broad use* . The nature of the changes ,to be made 
cannot, be anticipated, but will fall into the following categories: (1) 
content of the tutorial material, (2) diagnostic techniques and inter- . 
action witff users, (3) mechanical performance such. as computer response 
time or reliability, and (4) sysfepm coverage ~ the specific data bases 
and search services taught through, the system. ^ - 

Up^on:. revision. after initial use, the system will be vigorously 
publicized ^§nd offered, for use. in a wide area. Again, tl>e details will 
be developed in a later phase, but we must address* quesHons of how 
to publicize^ prove the value' of the system, price the system, and 
provide supporting services such as maintenance programming and instruc- 



- 0 - 



I 

'tlonal services. It is our expectation that the system will become 
self-supporting during this phase. 

Evaluation is not, in our -view, a single action. We are receiving 
some evaluative feedback at this time from use of our^ model system and' 
we will be performing other evaluations during the implementation anci 
initial use phases. All these, however, are formative evaluations. 
They serve primarily to help the systems designers to improve the system 
in some respect and to convince prospective users. of the system^s 
possibilities. Once IIDA is well into extended usage, it will be 
appropriate to perform, or to have an independent organization perform, 
an evaluative study of the effectiveness of IIDA in carrying out its 
original, primary objective: y the teaching and assistance in performing 
of bibliographic seanching./ 

Approximate planned timing of the major phases is: 



Design 

Implementation 
Initial Use 
Revision 
-Extended Use 
Evaluation 



July 1976 
July ^1977 
October 197S 
October 197,8 
October 1979 
1980 or beyond^ 



June 1977 
September 1978 
June 1979 
September 1979 
Indefinite 



1.5 Progress to Date 



r 



\ 



' The 'initial grant for Ph^se 1 of IIDA was awarde>i on June 10, 1976 
to cover a period of approximately twelve months. Th^ is the report of 
that activity. ^, * . | 

The products of the study are three-fold: a technical design, a 
plan for utilization and ^valuation', and a working model. The. technical 
design work falls into two broad areas: design of tHe instructional 
system and" design of the computer and data communications subsystem. :^ , 

The instructional subsystem encompasse.s the' teaching methodology, . 
design of st)ecific instructional material, design for exercise and 
assistance modes of operation which provide assistance to users, in 
conduct of actual searches, and diagnostics used to help sti4dents under- 
stand errors and omissions^ This subsystem^ is described in Section 2 of 
^this report. 

The-computer aiiji data communications subsystem is the^means of 
, implementing the instructional design. It includes computer hardware, 
software for the instructional and diagnostic programs and . software.' for 
the ^intricate linking of the IIDA^computer with the d^ta base computer. 
It is described in Section 3. \ !' 

Xhe plan for utilization and evaluation (Section 4) describes/ how 
we propose*to test the system^ evaluate test results, and make p^.ar\6 for 
the use of IIDA in real searching environments, whether in- a unix^'rsity ' 
or industrial setting. ^ 

The model of the system is a computer program, implemented oii a . 
PDP-10 computer, that performs a realistic example of every major IIDA 
function. Such a program was developed and linked to a data hasje system 
using a small file of air pollution bihliographip records. Tests were 
run and the data from these' tests used in preparing the final' design. 
The model and its use are -described in Section 5^ , 
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2. INSTRUCTIONAL SUBSYSTEM 



2.1 Teaching Methodology ^ 

'IIDA will offer a range of teaching materials' to Lhe student, with 
no int^ent that any one student must use all of it or that it be use4 
always in the same sequence. We assume our -users will differ too much'" 
in prior experience, computer familiarity knowledge of the data bases 
and experience or interest in searching for them all** to start at the 
-same\point and proceed in the^same fashion* Hence, the system will be 
designfe^d with maximuxn flexibility ior student control of sequencing. In 
doing this, ye recognize that we incur a certain risk that some people 
will try to work beyond their level of skill and will/b|?come confuse^ in 
doing, so^ However, completely preventing that would require too. much 
control over students. . ^ ^ ' ^ 

^It\ would be desirable to be able to set down thfe exact teaching and 
learriingX objectives of the IIDA system. Unicortunately, there is no 
accepted statement of what skills are required of a competent searcher. 
There simply is no way to make measurements of ' a student *s performance 
and pass judgment on whether he has or has not achieved some minimal 
level 9f competence. On the other hand, it is possible to make some 
measurements of his performance and to indicate, *in various categories, 
how well he performs in compar^ison to other students. Ov6r a long 
enough, period, these and similar measurements yet to b6 devised may 
evolve Into a true measure of skill in the task. To date, -searching 
remains an art whose success, is primarily self-judged: if the user is 
satisfied with results, cost and t'ime, it ois hard for an impartial 
observer to intrude and deny the satisfaction of the search. 

IIDA' s" primary task is to perform ih the assistance mode. We 
ascribe no particular merit to good performance in the tutorial mode if 
this is not followed by successful performance of actual searches'. In 
other words, a * grade* "in the tutorial material carries no weight except 
as diagnostic information. ' ' ' 

Ther4 are three modes of instruction: tutorial, toercise apd 
assist&ace. The terms instructional material or instructional program 
may refer to any of these.,. The tutorial mode provides the student back- 
ground material in the form- of cognitive information — facts ablsut the 
data base and search system. The .exercise mode represents the recpm- ' 
Aepded^ starting point" for learning performance. When operating. in this 
mode the student will perform' one or more actual searches', generally of 
this own choosing, but under control of the exercise program. He will be 
restricted in what commands he uses and how he uses them. " The primary 
purposes of the exercise mode are to give tlxer student some actual prac- 
tice in searching, tp get him to try e'ach of the cpiraaands at his dis- 
posal and for him to achieve success as early in his training as pos- 
sible. The assistance mode permits the student to work on his own 
search in an uninhibited manner. The system *s diagnostic facilities are 
available when needed. * ^ ' " ' ^ 

lb . ' 
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Students will be expected, normally, to wo^k through the tutorial 
and exfercise material as quickly as they can. They may remain in 
assistance mode as long as they like. There is no requirement that 
a student be 'weaned' of dependence on IIDA.* We do not expect that 
searching thi^ough IIDA will cost more than ^searching without its^aid, 
since our primary intended -user group comprises only occasional users, 
they are not likely |to retain all the- details of search mechanics from^ 
time to time. Hence, they may elect 'always to work through IIDA's , * 
assistance mode programs^ 

* These design decisions on the instructional material were made with 
a specific user group in mind. To repeat, these are working scientists 
and engineers, oi^ students in thefi^e fields. ^ They are not professional 
intermediaries. They are not necessarily the office 'gate keepers'' who 
function as infort|aal intermediaries. They are not frequent (say one 
^search per month or more) searchers. They are*not uncomfortable with* 
computers, although they may be uncomfortable with libraries. We 'do net 
wish to make the Claim "that this approach to training .is- best' for all 
classes 'of users. ' ' ' 




2.2 Tutorial Material ' . - / . 

Given unlimited time and -some way of requiring students to stari 
'with the tutorial material and worlc through' all of It before proceeding 
to the exercise and assistance modes, we would prefer i:o start, with a 
general appreciation' for the literature of the user's field. If, we 
could control the type of data communications terminal he uced, we would 
include training on how to use it. However, we cannot make either of 
these^assumptions. Hence, our approach is to teach the mifiimai mechan- ' 
ics needed, to allow hiin to by-pass some tutorial material, or all of it, 
but to retutn for ^more complete training as he begins actual searching 
and only then reali2es that there are some facts he would like to know 
but does hot. ; > , ' * - 

Most of the, tutorial courses are, of necessity, specific to a data 
base or search service or both. There is sttong sentiment in the pro- 
fession for a universal command language, a lingua franca of searching. 
We are in sympathy with this 'feeling but recognize the technical prob- 
lems of creating it as too imposing to be subsumed under this project.^ 
Our intent and obligation, therefore, are to teach the language^ and ^ 
d^ta bases that are there .and not to create new one^s. 

We 'will devote our initial efforts, ^to teaching students one data 
base through one search service.' Througfjput the implementation phase of ' 
the project we will restrict ourselves td a single search seinrice. 
Eventually, both the number of data bases\ taught and the number of 
search services taught, should be expanded. In expanding from one to 
many search services, it^may be possible tp switch to a universal ^ 
.language, if developed, to teach eacl^ language independently, or to use 
the ]^anguage of our first' ser/ic^ to search the others. The last approach 
probably implies some dimunition of^ service. For^ example, there is 
considerable difference in how the services search for strings or words 
embedded in a text.' Using ttfe language of" any one of them to command 
^any ot.her of them is likely to reduce the effectiveness of the second 
service. In the basic commands, for selecting terms, • combining sets, or 
browsing in retrieved records or a -thesaurus, there is little difference 
among the major languajges. 

19 . . 
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The following are 'the 'courses' that will be produced for .the fjtrst 
implementation of the full IIDA system: ' ' *)/ 

The search service to be used is DIAL'OG, offered by Lockheed 
Missiles and Space Company. The data base will be .Compendex, offered by 
the toginCsering Index, Inc. Each of these organizations will partici- 
pat,e in the design of teaching materials relevant to their own product. 

Below are the courses that will be produced for the first imple- * 
mentation of the full IIDA system. More complete course descriptions 
"are found in Appendix A. 

' Structure of Compendex . This course coy^rs material in the file^ 
the method of indexing and abstracting, the stru'ctui^e of the biblio- 
graphic^ record. As the first course in data base structure, this will 
•'cover what is meant , by data^base structure as well as giving information 
about Compendex.' , • > \ 

Search mechanics . This course will teach the DIALCXJ command 
language.^. 'Insofar as it is necessary or desirable to customiz'e the 
course for the data base being taught, it will emphasize searching of ^ 
Compendex. The course will cover all commands available through DIALOG 
and primarily teaches the syntax of commands and the results the com- 
nandk produce. This course will be in two parts, teaching first a,'^ 
subset of the language, then the remainder of the commands. ^, 

Search strategy . This course emphasizes how commands should be 
used, rather than what results each ^ohe produces. It teaches the 
student how to vi,ew his search as a goal-oriented task that^ requires 
preparation and planning and for which some progress measures are pos- 
' sible^ To the greatest extent- possible this will be taught indepen- - , 
dently of the specific search « service or data base in use. Probably 
"students will take this cpurse after having done some searching* 

• IIDA diagnostics . This course' reviews the diagnostic facilities 
avfiilable to the student user of IIDA. .This course, too, might be more 
profitably taken after the student has had some actual searching. ex- 

• perience. ' A clear distinction must be drawn in this course, between 
IIDA diagnostics and those provided by the search service, so the 
student will be able to work independently If he chooses -to begin 
searching without IIDA assistance.- 

, All of the foregoing inateriai is administered as c^onventional 
compute?: assisted instruction. Records of student performance, may be 
stored but there is no meaning imputed to performance at this level, nor 
would the results ,of th|.s part of the instruction ever be reported to an 
instructor who is girading student performance in searching as part of 

* some other effort, such' as preparing a term paper." 

\ 

1»3 ' Exercise Mode 

The purposes of the exercise mode are to: (1)^ introduce the .begin- 
ning student jto actual searching, (2) virtually guarantee success in at 
least a simjile search and (3) haVe^the student experience the use of 
eaih of" the commands available to' him. Although the exercl/se mode 
program permits the student to define his own search, it i^ still an 
instructional program. The student's attention is focusseci on how to 
use commands rath^;? than on the completion of a search. Students re- 
turning to IIDA after a prolonged absence might use this; exercise as a* 
refresher training co.urse. In eithei^ case the student muat be aware 
that this course is hot intended to conduct a search in the fastest 
possible way — that it is primarily* trying to teach him the use of the 
commands. 
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Ttie exercise will be divided into three parts: (1) a minimal/ 
* canned* search involving the use of only a few comands, to enable the 
student to see- quickly how a complete search is done and to give him the 
feeling of having stepped successfully through the- entirety of the 
process, (2) a full exercise using only a subs'et of the commaad language 
to enable himj:q learn the basic commands in a practical context without 
getting bogged douii in. too many commands, and. (3) a full -exercise in- 
volving all^comman^s in the repertoire.' The last of these might be made 
optional. , - ^' ' 

* i. 

2.3.1 The general design of^an exercise . 

The exercises are designed to provide a transition from the tu- * 
torial format to the search format. Ari exercise resembles a search 
although^. it is designed to be a teaching tool, not a searching device* 
Part of the objective of an exercise, especially the minimal exercise, 
is t6 give the gtudent a feiallng of accomfJlishment by getting him 
through a search as quickly and painlessly as possible. 

In ^appearance-, the three l^eyels of exercise all mimic a real ' ' 
search/ Behind this appearance of f reedqm to search at 'will are the 
resources, of the diagnostic programs.- At each level of the'jexercls.e ^ 
diagnostic programs are functioning. The; /HELP function ds aval^afile to 
the 'student at all levels of the exercise. This function can h^e^-'Valled 
by the student ^whenever he feels the need for assistance (details are 
given in 2.4), The perfortnance analysis routine (PAR) is- alert to" 
unsatisfactory \trerids of tnudent input and calls the /HELP function , for 
the student when' deemed necessary. Details of the PAR ^re given in 
2.6.3^ , . ^ ' 

The exercises introduce search language and jSearch^-strategy in 
stepwise fashion. Each exercise is complete, and- self -^contained. • At any 
time the student can prbceed to the assistance mode where the purpose is 
searching not teaching, and the program" becomes transparent except when 
/HELP is called. Program control becomes less visible as student search 
skills improve. The program structure' of the* exercises is discussed in 
3.~2. ^ ^ - ' 

The api)arent ^structure of the exercises is based on what little 
information is available on real-world searching. That information 
siiggests that searching can be regarded as a series of phases or states 
(see 1.2). These^are:' thesaurus search- and set *f ormation, logical 
combination of sets, and print of retrieved citations. The DIALOG^ ^ 
search language commands that belong to each phase are'; * 

Phase 1. SELECT and EXPAND, the* latter^ used both to display 

alphabetically related terms and 'terms related in meaning. 

Phase 2. COMBINE, in varying forms and, formats, and SELECT 

when used with multiple descriptors. 

Phase 3. PRINT. ^ 

These 'phases are not strictly serial in the* real world, nor are ' 
they ^ completely clear cut. However, It was decided to structure the 
exercise mode along the lines of the three phase search so that some 
structure could be imposed for ^teaching purposes. ^ j y 

"the student Is required to start In Phase 1 and to use all of the 
search coiipands associated with thesaurus isearching and -set fprmation. 
The formats of ^SELECT and EXPAND vary with the level of the exercise. • 

When the student has successfully tried all of. these commands and 
wishes .to continue to Phase 2, he is informed that he is nox^ able to nise 
Only Phase 2 commands. The formats and exact commands vary with the ■ 

. 21 ^ 



level of ^the^^exercise. TliesQ include the COMBINE cohnnand to form-s^ts 
by logical cdftbinations "of previously created sets and the SELECT in its 
text search form, allowing multi-descriptor commands. Again, use of 
phase 2 commands i6 requl;red before the student can proceed to phase 3. 

We recognize .the flexibility of on-line searching as being one of 
its major benefits, so in all but the minimal exercise, branching 
possibilities are presented when the student finishes phase 2, and can 
go 6n to 3 or- back to 1. A menu is provided to choose from, although 
the choices are no different from those choices available ^at any time fo 
an on-line searcher. The choices presented alXow the stu^dent to pass 
[back ^to another phase, to • continue' to* the next phase, or to ask for 
help. The requirement of choosing which phase he wishes to be in works 
to emphasize the planning required- in searching. 

• If thee student chooses to go back' to a phase already experienced, 
the controls on the required use of each command .are relaxed. The 
commands belonging to eat:h phase must still be u^'d together, i.e. he 
must finish using the single-descriptor SELECT before using COMBINE 
which is in another phase. However,/ it is no longejc necessary to use 
all command types in 'a phase. , . . 

If the, student elects to pVoceed with printlLng citations from sets 
already formed, a relevance judgement is required after ^ach citation is 
printed. We are stressing the importance of self-evaJ.u^tion of the 
results of a search. 

After viewing some citations," another branching, point is reached.^ 
The student can return to phase 1 or phase 2, he can declare him'self 
done with the search, or he can request /HELP. If .he declares himself 
done, he is asked to evaluate ,the citations retrieved to emphasize again 
the need 'for* searcher evaluation of results. 

2*3.2 Implement at io'n of the exercise mode 

4. Minimal exercise .. This -exercise is intended as a bridge from 
the quest ion- answer format oX the tutorial material to the command 
format of an actual search and to get the student through a ^complete 
search -as quickly and painlessly as possible. To illustrate the entire 
search process from beginning to, end, a short search is laid out for the 
student in the minimal exercise. The student docs the actual input of 
commands in the order of a real search, but' the' commands , descriptors 
and order of input are all. prescribed by the program. 

The commands available in the minimal exercise are- * . | 

SELECT using a descriptor 

COMBINE using AND(*) logic apd "two set numbers 
PRINT using tfi^ format 2, which -shows bibliographic informa- 
tion as well as indexing data. Format 5- of the PRINT command is 
.explained as including format 2 plus th * -abstract, but its use is 
Aot required. 

• The minimal exercise is patterned after the first search taught by 
the National Library of Medicine *s MEDLEARN system. ► 

' 2. The subset exercise . The subset exercise allows ^the student 
more freedom in searching, more commands to search with, and inor^i* 
strategy for searching. The student picks his own search topic. The 
concept of search phases is introduced in this exercise as a structure 
fot 'the student to hang his commands on. The three phases of Selecting 
terms, combining. sets, and printing citatidns are required to be used in 
this order to familiarize the student .with all the phases and to allow 
the student to see that. the phase structure really leads to a productive 
search. t 



The commai^ds available in the subset exercise include those avail- 
able in this minimal exercise with the addition of EXPAND. The SELECT ^ 
command will be usable with a reference number frpm an EXPAtID display as 
wefl as with a descriptor, but text searching will still not be allowed. 
The .COMBINE command can be used with AND, OR. and NOT logic. . Two new 
print formats, 1 and 6 are introduced allowing for printing of only 
accession numbers and of titles with accession numbers. The program 
controls when the commands may be used and. requires the use of all the 
commands available. 

In keeping with the attempt to simulate an interactive search^ 
branching is possible within this exercise. When phase 2 is completed, 
the searcher may return to phase 1. When all three phases have been 
experienced, the searcher is free to use the phases in any order. 

3. The full command language exercise . With the full exercise, **' 
the^search langua^ge completely presented. Sophisticated search 
techniques such as full text gearching^are made available and an attempt 
to use e^ch'^of them\is required. The^-phase structure of the search is 
maintained^ until all\ phases have been utilized once, then the searcher 
is free to move frotn' phase^ to phase. The control over the search is 
still divided as, in the subset exercise. The student chooses the 
subject and the descriptors, while the order in which the commands may 
be entered and the required use of all the commands are under the con- 
trol of the exercise. 



2;4 Student Controls and /HELP 

The IIDA system provides several student cpntrols which may be 
exercised any. time the system expects an input. Their purppse Is to 
secure assistance in searching, permit control over program sequence,, 
and allow communication^ to" other tepfiinals. All of the commands begin 
With a slash (/) to clearly distinguish them from DIALOG commands* The 
commands are /HELP, /GOTO,' /QUIT, /SEND, /DONE and /MORE. 

2.4:1 /HELP ' ' 

Jhe /HELP command calls an "assistance program that operates in 
conjunction with the exercise and assistance ^modes. Its purpose is to 
present descriptions of se^arch and system* commands and specific infor- 
mation on the search In progress. In its first aspect, /HELP can be 
used tc review the format and purpose of all legal search and system 
commands, and thus may be used in lieu of much of the tutorial material 
by computer-experienced students. Jn its second aspect*, /HELP provides 
data on the search in progress and tries to elicit ideas from the 
student on problem identification and solution. 

In response to the' /HELP command, a menu of tjrpes of help is 
printed The searcher picks which type is most appropriate to his 
needs. The help menu includes the following: 

1. Definition of search commands * The purpose and format of 
all .available search commands can be reviewed, one at a 
time. ^ , 

2. Definition of system (/) controls . The purpose and format 
of all system controls can be reViewed, one at a time. * 

3. Commands and search phase ^review .\ All valid commands used 

in the search so far are printed with the associated descriptor 
in the order entered. Additional data is provided on phases 
associated with the commaftds. 

2.-< 



Review of s^ts created . The sets created far are presented' 
in tabular form with. set number, number of Items In the set, 
arid creating command. If the command was 'a 'COMBINE, the 
constituent sets of the 'COMBINE command are translated into . 
their definitions in terms of descriptors. 

5. Review of sets already viewed . Sets vieWed through the use^, 
of the PRINT command are reviewed in a t^able which shows the 
total number of records in the set, the range of records 

' viewed, the format and the average relevance assigned to the 
range when viewed. If more than one range of records ha's 
been viewed for any set, each is shown. 

6. Review of descrit)tors used * All descriptors used so far in 
the search are listed together with the type of command of 
which they were the argument . • . ' * ^ 

2.4.2 /GOTO 

The major use of /cdTO is in the exercise^ mode where/ the command 
may be entered onljr^ after having passed tHrough all three phases at 
least" once. The /GOTO command must be followed by the number ly 2 or 3 ^ 
which indicates: 

/GOTO phase 1 (that part of the course where SELECT, and EXPAND 

may be entered) . ^ ' ' 

/GOTO phase 2 (that part of the course where COMBINE and commands 

of similar ef f iect may be entered) . , - ^ ^ 

/GOTO phas^ 3 (that part of the course where PRINT commands may be ^ 

entered. ^ ^ 

In the assistance mode, /GOTO is meaningful only as a memory aid 
since the system has no control over search sequence.' In this case, a ^ 
prompting message would be returned reviewing which commands belong. to 
the phase requested. 

2.4.3 7QUIT " . 

The /QUIT command terminates the operation of the IIDA program; 
All 4ata about the search is cleared and cpntrol is returned to a 
reception program which offers the 'options of logging off or logging in 
to any IIDA program including the one ju.st left. Users may /QUIT 
temporarily, in which case all results are saved until the search is 
resumed . 

2.4.4 /SEND 

The /SEND command initiates' a communication procedure^ enabling the 
student to send a message to the prbctor (2.7.4) » 

2.4.5 /DONE / 

The /DONE command denotes the' completion of a phase or completion 
of a use of /HELP. 

2.4.6 /MORE • ' 

The /MORE command is a signal by* the student of interest in further 
information on the subject under discussion. /MORE can be used only 
when listed as a possible response. 
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2.5 Assistance Mode 

The objective of the assistance mode of IIDA is to enable students 
to perform actual searches of their own choosing, in their owricmanner, ' . , 
while being provided assistance whenever needed. Unlike the exercise 
mode, assistance mode allocs the student to use ^any search commands in 
any sequence. He is not constrained by search phases and need not make 
use of ^ all the commands if he elect;s not to. The assi'jstance mode *pro- 
grams monitor the student's performance' and are ready to respond wicK 
diagnostic or other assistance whenever needed Need is determined 
either by the student or th^ program* The student can invoke, asj^istance 
with the command /HELP or, if the program determines that his performance 
indicates any of a variety of error j)atterns or lack of progress toward 
a successful conclusion, the program can invoke the /HELP facilities . 
itself;. . ' = . ^ ' 

We are again constrained by the lack^ o£ formal definition of how to 
proceeld in a search. If there were an algorithm by/^ich IIDA could 
determine exactly what the student should do, next, presumably it could 
do it for him. Thus, .we are dependent upon a .set^ of empirically and 
intuitively derived diagnostic procedures. The .intuition, however, is 
guided by extensive experience in teaching new users how to search and 
in observing the -kinds of errors and" misunderstandings that arise.. One 
of the long term outgrowths of this project;'. we fiope^ will-'be a defin- ' 
itive study of the measurement of progress of a search, or of degree of 
success, of of some related measure. • 

The assistance mode is the heart of the IIDA system. It is not, 
primarily a tutorial program. The student *s attention is intended to be 
.focussed on achieving his search objective, not search mechanics. The 
system should intrude as little as possible, consistent with its objec*^ 
tive of helping the student when he needs help. The various thresholds^ 
to be used to decide when. the system will intrude will have^ to *be 
adjusted with experience, as we learn both how much help students need 
and how much intrusion they will tolerate, as well as how much frustra- 
tion they will tolerate without active help* from the system. 
, Mechanically, the' assistance mode is- simpler than the exercise • 
mode. It does not control choice of command or phase of -search^ It 
does, however, make use of st performance analysis .routine . (PAR) which 
determines w^n tb interrupt a student. This pro^rap ,is based upon 
analysis of .the student history data maintained for, and used by, the ^ 
/HELP program. The^performance analysis routine is described in more 
detail in Section 2.^6.3 and 3.3.4. * 

In general, the assistance mode program will allow the student to 
use any command he desires* Syntax errors, occurring singly, .can be 
corrected..as quickly as thg student can be made to understand his error 
and re-enter his command. The PAR will perform a series of threshold 
checks after each student input of a command. When PAR determines the 
likelihood 6f a serious problem, it triggers a series of. more detailed, 
^analyses which are not performed after each command in the Interest of , 
saving time. When PAR< determines the* nature of the student's problem, 
it informs hlm^and provides the appropriate diagnostic data, then offers 
other diagnostic assistance through /HELP. Thus, the assistance mode 
program is primarily a ^ata recorder, augmented by PAR and /HELP. 

''When the student logs onto the IIDA system the grgund rules are 
briefly discussed: any subject covered by tjie data base may be searched, 
any commands in any order may be used, assistance is available on call 



by use of /HELP and assistance will be provided if IIDA feels the search 
is not progressing as it should. . . . . 

The student is asked to indicate whether his search objective is to 
retrieve a few very pertinent citations (1-10) , a short biblibgraphy on 
his, topic (10-25 citations), or an in-depth bibliography of the contents 
of the data base on the subje^ct of the search (more than 25 citations). 
It will be pointed out tnat the student -'is in no va^ bound to this^ ' 
choice. The data is used to make the perfomance, atvalysis routine 
function more adcguately by having *'an idea of the student's quantitative 
goal for the search. The methods for searching for these three types of 
result could differ, a fact which could be confirmed by<,the way in whiqh 
students use the system. In any case, th6 projected result of the 
search is of use in the PAR calculations. > 

Followitlg 'this interchange with the student, the prtigram disappears 
--£r.om the student's point of view. The reappearance of the program ^.is 
conditional on the nee^is of the student. If .he feels no need for assis- 
•tance, he may never cell the /HELP routine. However, the PAR continues 
to monitor his search. The PAR operates 'by checking thresholds as 
explained in 2.6.3. Each student input is checked against these thresh- 
oljds. Since the number of thresholds has purposely been kept to a 
minimum, there should be no effect on search time as far as the student 
can tell. "If the PAR discovers that a threshold has been crossed, an < 
in-depth analysis of the area monitored by the threshold is-. initiated; 
At this point, the student is notified that the search may be going less 
well than it could. The PAR in-depth routines are capable of locating 
problem areas but not in all cases suggesting remedies. The probrem is 
presented to the student and. his own analysis is invited (for details 
see 2.6.3 and 3.3.4). 

When the student and the program are satisfied that the direction 
of the search has been reconsidered, the student is returned to the 
search. Possibly a short review of hiS' most recent searching activity 
before the PAR was invokjj^is provided. This is dependent on the type 
of assistance needed whe^^K^he PAR is called. 



2.,. 6 Diagnostics 

The general IIDA approach to diagnostics consists of: (1)- re- 
cognition of syntactic errors in commands entered by the student, (2) 
detection of procedural errors (these are violations of rules imposed 
for pedagogical purposes only) , and (3) recording and analysis of 
patterns of usage. . ^ \* ' 

Search services are normally quite lacoAic in responding to syntax 
errors in commands. A typical response is 'Invalid command,' which 
gives the student little to work with when he tries to understand in 
what way his command was> invalid and what he has to^do ahput it. The 
opposite extreme in syntatic analysis is to try to pin-point an errpr 
using as evidence only the user's input, not his intensions. Here is a 
typical example. Suppose, the user has entered the search statement. 

SELECTION 

This could have either of the following Wanings:. 

SELECT ION 

S ELECTION CS' being the valid DIALOG abbre-* 
viation for SELECT) 



or he may have meant but omitted to prefix a command abbreviation to 

give: * ' . ^ " . ■ : 

9 "S SELECTION ^ , ^ ' 

^ ' or " , SELECTION , — ^ . 

Given only the character string SELECTION, there is no way a program *^. 
can tell what was^ intended . An attempt to do sO' may be more misleading 
than the unhelpful "invalid command." For example, treating it as 
SELECT ION (which DIALOG does)^ returns a^set number, count; and descrip- 
tion which can be quite confusing if _"E SELECTION^' was the intent.^ 

Hence, our objective is to give^ more infbimiation than -merely that 
an error was made but^ to try to avoid being misieadingly specific about 
the cause. IIDA performs **:s own parsing* of user commands for two 
reasons^ First, it stores data about the nature of erroifs made for 
later analysis. Second, some pars-lug is'imcessary in" the concentrator 
program (See Section 3.6.4).* Siu^e the work has to be done., all useful 
information might ag well be extracted at one time. ' 

Procedural errors are encowtered only in the exercise mode of 
IIDA, which restricts the ^student to us'e of ' certain commands at certain 
times., requiring .that he use ohly those commands authorized during any 
given phase of* a search. * He' Q^tinot, for example use a print command in 
phase «1. Detection of procedural errors is" a simple matter of looking 
up each i\icbming command. in a^'list. The list to be searched is a func- 
tion o£-the phase of .the search. - ' ^ 

►Analysis of patterns, of^usage ponstitutes the bulk of the diagnos- 
tic work^of IIDA.^ This analysis is performed at four levels: (1) 
collecting basic performance data fropi each student, (2) microanalyses 
of data elements, (3) threshold checfeing, and (4) overall assessment of 
performance. - ^ < ^ / ' • 

2.6.1 Diagnostic data , , 

^The most fundamental data t'o be recorded is a 'simple record of the 
commands entered by the studfent and the^ responses to these sent from t|ie 
data base processor'. To this' is added t;he results of some of the 
microanalyses. The datfa aye classified, as to purppse and recorded in 
the various history files. Each file represents a record of student 
performance from a different points of viw and they are somewhat re-^ 
dundant with each other. /The purppse of this organization is to enable 
the student proctor , or diagnostic! program view the student's per- 
formance from different perspectives. ; Some ^udents may prefer to 
review cheir own- work in tertds of sets created. Others may prefer to 
see the .sequence of commands issuedJ Still others imay be concerned only 
with the records retrieved. 

This diagnostic -data is available to the searcher in response to 
the /HELt command and is available to be used by the PAR. The PAR uses 
this data to decide whether a problem situation has arisen and uses the 
daca again to present the problem to the s*tudent and to help the student 
resclva the problem.' . 

The eight types of data to be maintained are: , (1) command history 
(2) set hisdDry (3) sets viewed history' (4) descriptors used history ,(5) 
error history (6) /HELP history (7) records viewed history (8) expc^aded^ 
index data. Typical examples of the kinds of data maintained are listed 

"here, for detailed inforiiiation on contents and maintenance of data files 

* see 3.5. \ 

■ . \.^' . ^ 
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!• CommanH history '\ ' 

Text of connnands 

Search phase for commands 

Command type code %" 

Time data ' . \ 

2. Set history ' 

Set number \ . ** 

Type of command creating set • ' "i . 

Number of items in set / - 

Expansion of set description for those sets defined in 

terms of other set numbers, the nutobers being replaced 
by the descriptors generating the original set. 
. Cluster number (see 2. 6.2) 

3. Sets viewed history " , ^ 
Set number ' • 

Which records of set viewed^ 
Format in which viewed 
Relevance of records^^. 

4* Descriptors used history - » 
Text^of descriptor 

Number of times used as argument to SELE'CE, EXPAND, COMBINE 
.5 Number of records indexed under it * 
. Number of related terms 

5* Error history \ * ^ » 

Text of error 
. Code* for type of error 

6. /HELP history 

Number of times each kind of ^ help used 
Location of /HELP call in search 

7. Records viewed hi ^tory 
Record accession number 
S^t from which record came 
Record number in that set 

Format in which viewed ' ' . . 

Relevance assigned at that viewing 

Above is replicated for each time record viewed, whether 
as member of same or different set. 

8. Expanded index data 

First and last terms in every EXPAND table generated in 

this eoarch <used to be able to tell whether any given 
descriptor descriptor was seen in an EXPAND table) . 

Files 1-4^ or perhaps 5 will be available to the -searcher through 
the /HELP command. All of the files are used by the PAR for diagnosti 
purposes. 



2«6«2 Microanalyses 

The microanalytic programs are those needed to produce the gletaents 
of the history files which are not mere listing of student or DBF inputs. 
The following programs ar^ required ♦ 

1* Command parser > Purpose: to verify that student commands are 
syntactically correct and procedurally 'acceptable. Functioning: the 
program is given a list of expected elements by ^the instructional pro-* 
gram that calls or invokes the parser (i,e, by the tutorial, exercise or 
assistant program) • The calling program might specil/ that any of a set 
of commands is acceptable, with any legal argument (as a SELECT or 
EXPAND command with any descriptor). The parser determines if the 
required conditions are met and, if not, records the defects. The 
parser than converses with the student, informing him of the nature of 
the error and eliciting from him a, new response (i,e. new command). 
Responsibility rests with the parser to get a correct command from the 
student. The parser maintains a record of errors made. See Section 
3.3.1 for more detail. 

2. Phase detector . Purpose: to determine in what phase ^of a. 
search the student is working, recognizing that there are^^not neces- 
sarily sharp lines of demarcation between phases once beyond the ex- 
ercise mode. (This program is of use only in assistance mode: the 
exercise mode controls the student*s phase for him), functioning: The 
problem is this: given a string of commands, there are embedded within 
it sequences of "like" commands, i.e. commands all L^elonging to a given 
phase. , The difficulty is that these sequences may not be pure, there 
may be a phase 2 command amid the phase l*s» The phase detector must be 
able to recognize one or a small number of out-of,-phase commands in a 
sequt..*ce as not neces'sarily breaking the sequence. For example the . 
sequence,. 1112222233 clearly divides into discrete phases In the ' I 
sequen::e 11121122 we might say that all of 111211 is^ f rom phase 1, in 
spite of an out of phase command.^ 

3. S,et description expansion . Purpose: .to assist the student by 
* displaying descriptions of sets he has created previously in terms of 

descriptors rather than set numbers. Functioning: *A record is kept of 
the argument of each command which creates a new set. When a COMBINE 
command is added to the data file, thevset numbers used* as the argument 
are traced back to the command which created each of the sets. If the 
argument to that command is^another group of set numbers, they are 
traced back 'to their creating commands. At some point, the creating 
command will be a SELECT command. The argument of that command will 
replace the set number in the description of the command using that set 
number. The descriptors replace set numbers up to the command which 
initiated the tr^lce. The COMBINE argument with elementary set numbers 
and the argument with terms replacing set numbers are both stored for 
each set formed with a COMBINE command. 

4; Set description clustering . Purpose: As complex sets are 
created out of elemental sets (those defined by a single descriptor), 
some of these new sets will closely resemble others in composition and 
some not. The clustering algorithm will relate together those sets that 
use * similar* descriptors in their definitions. It detects the point at 
which a student departs from work on a previous cluster and begin^ a new 
one and .,it measures the relationships among clusters. Functioning: A 
measure of similarity S(i,j) is used to compare the most recent command 
(C^) with a previous one (C.). If the measure of similarity is high 
enough, the new command is included in the same cluster as the previous 
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one. If>j[iot, a new cluster is created. Relations between clusters are 
measured by l i^ks , A link exists between two clusters if a comroand in 
one cluster has a high similari^ty measure to a command in another cluster. 
The measure between the clusters is the number pf such links. Ififor- 
»natlon on the- point at which a .student makes "the transition to a new 
cluster and the relationship among clusters is proviiied to the performance 
analysis prograiTi. The clustering program is described in more detail in 
Section 3.3.1. , ' ' ^ 

5. Descriptor viewing status . Purpose: this program, will de- 
termine whether a given descriptor has ever been seen by the student, 
during »this search, as part of an EXPAND display, irrespective of 
whether it was the search term in the display request or merely .appeared 
in response to a search on another term. The program will make a dis- ^ 
tinction between being viewed in an EXPAND [termj^ display or a sub- 
sequent EXPAND display showing related terms^ rather than alphabetical, 
neighbors -of the search term. Functioning: When a command has been 
identified as EXPAND [term] in the process Of updating the data file,. " 
the 'f IrL'. and last terms of the table generated are recorded on the 
expanded index data file, in alphabetical order by first term. When {7 
requested, the descriptor viewing status program checks the expanded 
index data file to see if the term in question falls between any of the 
pairs of terms li£:i:ed there. 

Some of these* analytical programs can consume significant ^amounts 
of computer time. It is necessary, therefore, to carefully ration the ' 
frequency with which they are run. Ji clustering algorithm, for exam- 
ple, may be run only when the set history display is requested as" part 
of /HELP. It probably- (subject. to timing estimates when actually pro- 
grammed) should not be run after each command. 

2.6.3 Performance analysis 

The performance analysis routine (PAR) operates in conjunction with 
the assistance mode of IIDA. Its purpose is .to detect difficulties or 
problems in ihe conduct of a search that might lead to failure. It does 
no t^ look for specific syntactic or procedural errors. It looks for 
adverse trends. A well trained, highly perceptive student could operate 
without PAR, because he would be able to. detect his own difficulties and 
su&mon the diagnostic facilities of /HELP to get specific information 
and ideas on proceeding. PAR is intended to make that initial decision 
for the student — the decision to call'*for /HELPi 

In order to avoid excessive delays in resppnding to students, it is , 
necessary to avoid lengthy calculations or processes following each 
student input. The approach to be taken, therefore, will be to have the 
PAR make some rapid checks to ^ee if there are any indications of prob- 
lems and then to make detailed analyses only when the preliminary checks 
so indicate. 

; Preliminary Analysis . There are five preliminary analysis 
subroutines which are used to make rapid checks of student performance. 
These are: (1) phase an.d cluster analysis, (2) zero set formation, (3) 
use of extraneous and redundant commands, (4) use of '/HELP and delays in 
responding, > and (5) error analysis. The purpose of each subroutine is 
to quickly check one or more threshold values to determine if there is 
^•indication of a serious problem. Since th^se checks; are run after 
each student command, only a limited amount of time can be devoted to . 
them if we are not to delay the student excessively. The exact thresh- 
old values can only be determined on the basis of experience. In the 
remainder of this discussion- values are occasionally mentioned, but 



these are illustrative only ~ 'all are subject to change. Upon detec-^ 
tion of a .{)Ossible serious error, more detailed" aiialytic programs are ^ 
called and it is these that develop the data 'presented to the student in 
discussions of errors with him. 

Threshold values^may vary with the student *s progression through 
search. During, the first, few connnands, thresholds may be set quite low, 
"because too matiiy. errors here may indicate an unprepared student who is 
likely to produce only chaos if allowed to proceed.. On^ the^-other hand, 
once he gets successfully underway, the thresholds can'' be^ relaxed to 
permit him an occasional error ^'or an unsuccessful excursion into some 
particular subsearch. If the total number of commands issued exceeds 
some number (which will be base^d on our own and others' experience in - 
monitoring searches) the thresholds may again be lowered on the ass^ump- 
tion that the stfudent is unable to bring the search to a conclusion arid 
will need hel^in doing so. We use^the^tferm convergence to indicate 
that the .number of elements in *a series of s'imildrly-def ined sets 
(cluster) is tending toward a numbef consistent with 'the student's 
initial search goal. Failure to converge may indicate that the student 
searcher is*trying to be ovenneticulous or is expecting the system to 
prpduce results' it is not capable of. We^'use the term- thrashing to 
indicate the opposite b'ehavior of excessive concern with oi;e set: a 
'rapiji switching from one set definition to another, "dissimilar one, 
without takingt the time to adequately explore the earlier possibilities. 
Preliminary analysis is concerned with spotting either of thes§ trends. 

The five preliminary analysis subroutines are (described below. 

1. Phase and cluster analysis . 

Together, these provide' a quick analysis of the pverall » 
,trend of a search. They can detect failure to converge as well a^^ 
thrashing, in checking such values as: number of clu^sters formed, 
number of commands issued per cluster,, number of commands per 
phase, number of phase changes, number of records. per set within a 
cluster (i.e., tende;ncy to converge within a cluster). 

0 

2 Zero set formation ^ 

A zero .set 'is 'one with no records in it. While zero' sets 
are not always ^an indication of error^ as in fine grain sear^:hing 
of patent files, an "excessive" number is an error indication, A 
student will be able to inform IIDA that his zero sets were inten- 
tional or knowing, not the result of error. However, the 'zero set 
•checks can be valuable in detecting misuse of commands, lack of- 
understanding of boolean logic or attempts to be overspecific in 
set description. 

. 3 . Extraneous and redundant commands . ' ' « 

Except for commands such as PRINT or DISPLAY SETS, any 
repetition of a command witfiin a search is normally" unnecessary. 
In a lengthy search it is not necessarily a serious 'problem,- but 
Over-repetition, especially the immediate repetition of *a command, 
can indicate lack of understanding of the commands or of how to use 
IIDA's or dialog's facilities for reviewing past perf oirmance. A 
not uncommon 'error among beginneris is simply to r^epeat a command 
that has led to an unwanted result, or to repeat a command that, 
, because of communication delay or machine overload, has not yet 

been responded to. An example of an- extraneou^^ command is a SELECT 



; (descriptor) which produces a noh^-zero^^et , but whose resulting set 

J, is nonetheless never used. Again as an occasional thing this is 
no problem, but if done too much it is an indication of lack of 
Rlauning on the part of the student.' 

, o " '4. / HELP and delay analysis . 

^ /^HELP and time use, «rhen excessive, are indicators of 

V searcher .problems. • Thresholds her^ are the number of calls for 

/HELP compared to the total number of commands, amount of time 
spent in /HELP, and the amount .cof time betwee^i 'commaads. When more 
time is spent using /HELP then doing searching, the student should 
probably go back to the tutorial material. Iti-depth analysis of 
^ the data jon /HELP, use will allow a reasonably accurate assessment 

of the kind of tutorial matierial needed by the student or perhaps 
recommend that the student call the proctor. Excessive time between 
commands can indicate that the student is unsure of what to do 7 - 
next, of that he is being distracted. 
. ^ ^ • 

5. Error analysis 

Error analysis examines the nimiber of syntactic or pro- 
cedural errors and computes the total nimiber of errors in the 
' - search, and the tota,l per last n^ commands issued. Early in the 

search, the threshold will he set low and any repetition of an 
error will draw comment. Later, the thresholci will be raised. It 
is the command parser that detects' errors *and conducts the initial 
conversation with the student about them. But the parser is limited 
to trying to get the error corrected. Error analysis becomes 
concerned when patterns of error repetition are detected. 

% 2. Detailed analysis . Whenever a threshold is crossed, detailed, 
in-depth analyses are called. Whenever^the detailed analysis program is 
called, some negotiation with the student will follow. The purpose of 
^ . the detailed analysis is then to provide the detail for this conver- 
sation. The kinds of analyses performed are described below. 
* I 1 

1. Phase and cluster analysis . ^' 

The detailed version of this program is given the com- » 
' • -Tplete history of the search and data on the' related thresholds 

'exceeded. If far enough into the seatch, measures of relatedness ^ 
among clusters are computed and this data used to indicate the 
pattern of the student *s use or lack of use ^bf results of early 4) 
clusters in formfng later ones. The student will be asked to 
indicate, in natural language, why he has taken the course^of 
action Ixe has. This information serves two purposes: it* makes him 
think carefully about what he Va^ doings and, when read and analyzed 
by the proctor, provides information about 'student behavior th^it ^ 
might not be available from^the programmed calculations, 

. 2. Zero set analysis . 

' ' c This timfe a fine grained analysis is made of ^all" zero 

sets, the command types and the search^ logic that led £0 them. If 

there is a pattern, the student is so informed (e.g. excessive/ ^ 

number of ANDs- in a COMBINE command). If a single-descriptor 

SELECT results in a zero set, a check will be jnade on whether that 

descriptor has appeared on axi EXPAND display. The student will be 

< * 
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reminded that he might have checked the term, that he seems to' be 
developing a pattern of use of unchecked terms ♦ He will also be 
given ^a list of the ni^ber of terms on a display centered! on his 
?:ero-set descriptors that have m out of n letters in common.' For ^ 
example, if he uses SELECT COMPUTABILITY and gets a zero set, IIDA 
will see if the term COMPUTABILITY had been seen by this student on 
> o . any EXPAND table. "If not, the program issues an EXPAND COMPUTABILIiy 
' and counts the number of words appearing on the resultant table ^ - 
that have a given fraction of its letters'^ in common with the search ' 
teral Clearly, letters in common dp not' imply equality' of meaning, 
but the student will make that decision. 

3. Extraneous and redundant commands. 

This program classifies and sorts the extraneous or re- ■ 
dundant commands. Fcfr example, it will point out repeated adjacent 
^ commands, failure to use the results of a particular type command, 

excessive use of a tyj^e command > etc. 

^ ^. /HELP and delay analysis . Data produced herein is for the 

proctor, not the student. If a student is- using /SeLP excessively 
or 'taking an unusual amount of tim^ between commands, it will 
probably- do no good to have the computer nag him about it* Data 
will be sent to the 'proctor who should then do his own analysis of 
the situation and communicate with the studenjt. ' Data given to the 
proctor will include analysis of how and how often /HELP is used 
\ ^ and in' what situations, if any pattern can be found. 

5. Error analysis * The preliminary error analysis program 
merely counts errors. The detailed analysis program classifies - " 
them and presents the result to the student. 

' 3. Result of use * When a threshold has •been crossed and the in- 

depth analysis has been run, the next step is negotiation with the 
student. The PAR is designed primarily to detect trends, not to de- 
/ termine the cause of errors, so the infoirmation from the in-depth 

analysis on which PAR based its decision is presented to the student or 
proctor for determination of cause. Some alternative remedies may be 
presented, depe..ding on the problem* The student is then directed to 
formulate his solution to thfe problem and indicate 'to the program when 
he is satisfied* It is at this point that^the student can tell the 
program that what the PAR has detected is not an error, butjiar't-'^f^^e 
student's strategy • The student evaluates hisse^rshT'''maices the changes 

' he desires and goes back to his searclu 

In the -case of the ^tudeivt-'X^KoTias no idea of how to proceed, the ^ 
options of calling fcf further data and assistance from /HELP or calling 
the proctot are always open. PAR will be helpful and' non-threatening to. 
the student. It is not a panacea, however, nor is any attempt made for 
It to recognize every type of problem. It; would hardly be cost effec- 
tive even to trx> so long as. a proctor is available and his involvement 
costs little. 

2^6.4 Measuring performance 

We have previously made the point that we cannot measure the ac- 
quired searching skill of a student in any absolute sense. There is no 
numerically defined measure of how well one searches, hence no way to 
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measure the degree to which any given individual has progressed toward 
that goal. However, there are some measures that can be made of the 
relative performance of students by comparison with one another. It is 
important not to attempt to use such Figures to assign grades to stu- 
dents or -to measure on-the-job performance in industry. The measures 
.can be meaningfully used to g^in insight into aggregate student per- 
formances, sequencing of training material, and value of diagnostics'. 
The following kinds of measures can be made. 

1; Performance in tutorial material . A record will be kept of the 
couVses taken by each student and the sequence in which they are taken. 
Performance can be measured by recording right, and wrong answers to 
questions. It should be noted that we have often written that IIDA does 
not involve the concept of 'right' and 'wrong' answers, but that applies 
to exercise and assistance modes. The tutorial materi?4 is not 'to be 
written to serve as a test of knowledge or skill and it is important to 
realize this. However, knowing* what kinds of errors 'students make Is of 
some value in assessing how well th^y have mastered the material and 
where the tutorial material is weakest. 

2* Performance in exercise mode . Records maintained here should 
indicate the kinds of errors detected and the manner of using the /HELP 
facilities. The exercise mode' program will offer three levels of in- 
struction: a very brief search designed to assure that the student can 
successfully negotiate it, a search using a minimally useful subset of 
commands, and a search using the full repertoire. Certainly in the 
second of. these it is expected that errors will bB'made, as is the case 
with the new commands introduced in the full-language, search. Thus, 'raw 
score is not too meaningful, but measures can be ma4e of Errors com- 
mitted and subsequent behavior relative to that error, i.e. whether the* 
student eventually stops making that type error. Since the emphasis in 
the exercise mode is to gain familiarity with the commands'^ rather than 
to retrieve useful information, not too much should be inferred from the 
student's reaction to the data actually retrieved at the conclusion of 
the search. 

' '3. • Performance- in assistance miode . When the student reaches, this 
point injjiis-training, a direct attempt to measure his success in the 
^arch^^s, appropriate. Counts of errors made, time spent using diag- 
nostics, number of commands and amount of time used can be useful, as 
will the student's own assessment of the relevance of the records he 
eventually retrieves. ' / 

Using evaluative information of the kinds outlined' above, we will 
surely see differences among students. This can be combined with in- 
terview data taken from seledted students, in which the interviewer 
attempts to ascertain the student's attitude toward his IIDA experie,:ce 
and the degree of success he feels he achieved. 



2i7 Operation of IIDA 

Each of the three major modes of use of IIDA — tutorial, exercise 
and assistance — has its own primary training objective. These are: 
(tuto^rial) to provide basic instruction on data base structure, search 
mechanics and strategy, (exercise) to provide experience in the use of 
the commands in carrying out a search, and (assistance) to assist users 
'in the performance of actual searches. In practice, the objective the 
student may* have and the benefits th.e student may derive will not be so 
cleanly separable. Also, the student tnay not view the change-over from 



mode to mode so distinctly. Students can accomplish search objectives 
in exercise»mode and can learn in assistance mode. -They can invoke 
tutorial material in assistance, mode when needed hence use tutorial 
material for reference while performing searches. The purpose of this 
section is to review' IIDA facilities 'from the point of view of the 
student. 

2.7.1 Mddes of student- use ^ ^ • 

The average novice user of IIDA should start his 'training with the 
data base structure course and also take the search mechanics and search 
strategy courses befdire proceeding to the exercise* The diagnos^tics 
course may be Xeft for later. It will be possible, however, for a 
student to. begin with any tutorial he wants, or even with the exercise 
mode program. A record will 'be kept of courses taken and their sequence 
If a student shows signs of difficulty in handling the more advanced 
work, he will be referred back to tutorials not taken. 

The search mechanics course will provide instruction on two levels: 
»the minimal subset and the remainder of the commands. Ic will be 
ppssible for an inexperienced searcher to take only the subset mechatlics 
and data base courses, and be ready for the first exercise within an 
hour. . * . - ' * 

-To summarize, students will be permitted to start at the beginning 
of * any course and proceed in any sequence. We will have a recommended 
sequence and we* will be able to suggest, under certain circumstances, 
that the student now divert himself to a specific course in order to 
help solve a problem in searching. " , , ^ • 

.Records of experience of students taking various paths through the 
teaching material will undoubtedly provide a basis for fiirther refining 
Dur suggestions to new students on their choice of -sequence. ^ " 

It will not be necessary for a student to take all his work in a 
single sitting at the terminal. Once a student sighs'on, hisr^ecdrd' ^ 
will be kept for a reasonable period and he may resume at any time. 

2.7.2 Student records 

For each student using IIDA, a record will^ be maintained of his use 
and performance. ' There will be a background record of each student, a 
record of performance in tutorial material, a detailed history of his 
performance in an exercise or assisted search, and summary information 
on exercises and searches previously completed. Specifically, the 
following information will be maintained; 
1. Background data . 
^ Student identification number M 

Major subject, academic discipline", or profession 
Time, since bachelor *s degree 
Self-assessment of computer experience^* 
Self-assessment of typing skill 
Self-assessment of library use skill 
Self -assessment of skill in on-line searching 
Total amount of time spent in tutorial, exercise 

and assistance modes. 

*. * 

' 2. Tutorial recor d. . 

Course or course segment 
Time and dat^ begun 
^ • ' Time and date completed 

■ / 



/ ' 

^ Percent of questions correctly answered. =k 
Specific questions answered erroneously, with text 
of wrong answers (Note: * While bourses can ' 
- only- be begun at the beginning; the' student 
sign off temporarily, hence complete 
only a segmenc rather than the complete 
course. Also, individual frames may be in- 
voked when a student is in exercise or as- 
sistance mode. Informatioiv on wrorig an- 
swers is for use by IIDX staff in analyz- 
ing, performance of the tutorial material, ^ 
not the student.) 

3. -Exercise ' - ' • * , n 

For each level of exerc^^se, the /complete set of 
history files will be main^tained. . 

For a completed^ exercise, a sunmiary of each history 
file for analysis* of perfprmance of aggre- 
gates of students. " j' 

4. Assistance, mode . / . 

Saine data as exercise mode. ' 

2.7.3 Privacy aspects of student records 

There is no need or desire to collect or retain any information 
of a private nature abou^ users of IIDA» HowWer, because of the great 
amount of sensitivity on the subject of privacy and because the initial 
use of IIDA will be in conjunction with -class, assignments, several 
positive st^ps will be'^ taken to assure privacy and to assure students 
that their privacy is being maintained. Tliese steps are: > 

' 1. |i positive statement will be made to the student, through 
his terminal, at the start of Ms first encounter with IIDA that: 
(1) A record of his performance Is being maintaine'd for research 
purposes, (2)^Thid record will. in na way^become a part of his- ' 
academic record nor ^will -it be used to determine. In* any part, his 
grade in the course through which he is using IIDA, (3) 'These 
records»will not be disclosed to^ his instructor and (4) Ris name^ 
^ is not a part of the record; instead his record identifies him/only 
by a number ,assigned by the IIDA staff solely for this purpose (not 
related .to employee, social security or student numbers, for example) 

p " 

^ 2» The, only record, of a student *s name will, be a master list, 

/not stored in- any computer, that relates students* names and ideti- 
tification numbers. Such a Itkt 'is necessary to assign numbers' and 
to remind students who forget them (a common occurrence). 

3* No identification of individual students ^ even by number, 
will be made in any published record of. IIDA activities or accomp- 
lishments. Records will be used^only to -study factors affecting 
performance and the only reason ever to publish part of an ipdlr'. 
vidual record would be i:o , illustrate a particular behavioral 
■pattern. Identification would not then be necess>ry. 

l^f:^ Proctor . 

The proctor is an IIDA staff member familiar with the various 
teaching programs, data ba^e searching, and the use of data-ccmmimi- 
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cations teniiinal3. The reason for having a proctor is to have an 
experienced person on call when unexpected problems arise or the com- 
pdter is^unable to resolve an issue. An' example 'of the need for a' 
proctor is when the student is, having trouble using the terminal and is* 
unable- to send a meaningful message. Then, either the proctor can see 
that the student Is having a problem and call him or the student can 
'Call the proctor on a voice telephone line. 

The proctor will also serve to collect data for revision of IIDA. 
Again, when unf6rsepn problems arise the human proctor may be the- only 
means by which the nature of the, problem can be communicated. 
. The proctor will have the following capabilities: " 

1. Student monitoring . T^e proctor will be able to request 
the IIDA computer^ to send to his terminal a copy of all messages 
into and out of a specif ied. student 's teminal." Thus, he will be 
able to monitor the student^s work.^ Whenever a student is being 
monitored, he will be so informed by a message from the computer. 

'2. Retrieve student performance data . By use of the /HELP 
command the proctor will be enabled to browse through the history 
records of any "student curre^ntly conducting a search. He will have 
access to the same data the ^ti^dent has, except that the proctor" 
can see* the student's /HELP history file^ but this is not offered 
to the student. * ^. 

— 3. Retrieve tsmmnary data . The proctor will be able-^to invoke 
previously written programs that will prd^ide summary information 
about all students. For example, he may ask for a report which 
includes a statement of how many students gave the wrong answer to ^ 
question 2C in a tutorial, or how often students make syntactic 
errors in PRINT commands. The proctor will not have a program 'that 
enables^him' to ask ad hoc questions of any file in the system*. 
This is 'simply too expensive a facility to provide. 

4. Typed messages . ■ Students will be able to type messages • 
for transmittal to the proctor's terminal, whether or not they are * 
being monitored, and the proctor will be able to send a message to 
any student terminal.' This capability can be used for asking and 
answering questions when the terminal is running and the* student 

• needs advice or the proctor feels he must tell somet':iing to the 
student, 

5. Voice telephone . The proctor will have a voice telephone 
line separate f r'om. the one used for his terminal. Students, may or 
may not have such an extra line at their locatidjns. At any rate," 
if the student can get to a telephone or can disconnect his digital 
terminal, he can ae^ to the proctor di-rectly, by voice. There will 
be no cost to the'^udent for this service. > In the future, when 
IIDA. reaches a Mftional audience, this free service should be 
continued as it ^is by the major search service contractors. 

Use of proctor assistance by a student will be considered a use of 
/HELP and will be logged 'in the /HELP history file. 



3M* 
i 




3; • CO>ffUTER\AND DATA C0>M0S1CATI0NS! SUBSYSTEM ' ■ 

3.1' Overview of the. Computer- System 

' 1 ^ ' _ , ' ^ ' 

Three major .computdng £\;^ctloa« are perfpraed within, the IIDA system; data 
base processing, instructional "l^rocessing azld communications^ processing. The 
computing system thi^t per^jforms »eacK of these functions is called a prbcessor * 
which term denotes a program running in a computer*. T;he three processors, 
in concept, could be all implemented in one physical computer or in thref 
•Separate ones. In the IIDA model (See Section 5) all thr.ee, were im^ilemeilted 
within a single computer. In the full IIDA system there will be two physical 
cbmputers, one operating the d&ta base processor and one operating the 
instructionsLb and cotmunications proclessors. o ^ 

The data base procesior (DBF) performs information retrieval functions 
and has access to the bibliographic data bases, including Compendex.. Initially^ 
c there will be a single data base processor, that operated by Lockheed 
for its DIALOG search service. Eventually, it. will be possible for IIDA 
to communicate with other data base prqcessors. IIDA students will use ' ' 
exactly the same service that is available to other\ direct tisers of the 
DIALOG service. That is, they ^ will have the full, up to data-data bases ' ' 
and they will even face problems of slow- response when *the data, base pro- . 
cessor is overloaded. . ^ * , 

The instructional processor (IP) is a set of prograM , which administer 
the computer-assisted instruction or tutorial training, the exercise and 
assistance modes, and provide, the diagnostic programs for use by these 
programs and the students. The instructional processor deals only with 
messages, given it by qther programs, and provides messages to be delivered 
to students or the dat& base processor. It is not concerned with the mechanics 
of message receipt or transmittal. 

r*.i;,r!5''?I^'.i,^^K of 'oessages can sometimes b* complex, all functions 

related to the handling of messages are combined and together comprise the 
r^S^S processor (CP). It serves as a central communlcSons facility 

""^^ processors and student terminals and 

delivering them to the appropriate' addressee, which is- one of the other- 

L™n?ai°Jl"1oTora!r:L^\S^^^ co»».nicatlons processor 

aTr«« -^os.or axx message traffic, serves as a concentrator to" 

tl rZ llv^ ^^^^f^""^ '° °" ^ single line between the CP > 

and the DBP, supports the proctor and computes the charges for all users. 

3.1.1 General flow of data _ ' 

-.f A ^^^*^f ^J"" illustrated in Figure 3.1.' Inputs may , originate with 
student terminals,, the proctor terminal, the Ijistructional processor or the 
t J,T P'^^^f'^J- All pass through the CP wherein they are tagged as 
to. origin and destination and put in the appropriate queue for delivery 

^ ■■ ■ 38 " . 
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Figure 3.1. General flow of data in the IIDA system. 
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to addressees. There may be more than one addressee when, for example, 
the. proctor monitors a student^s. performance, hence both receive DBF output. 
A log is kept of all communications transactions (arrival, enqueuing, de- 
queuing, transmittal) for later use in analyzing traffic patterns and delays. 
Messages to and from the DBF may be modified by the concentrator. Finally, 
costs ar.e accrued for each student using the system. Except for the fact • 
that the concentrator must parse student , commands a^does the IP,, in order . 
to be able\to modify them, there are no functions in common to > the IF .and 
CP. The .CP ^^hould be "transparent" to users, i.e. they should be unaware 
of its existence. , * 

\ 

\ • ^ • 

3.1.2. Hardware tronfiguration 

■■ ' \ ' ■ 

The hardware configuration used as the basis fpr design. is the smallest 
Digital Equipment Corporation FDF-11 that could ^andle the IIDA job for the 
forseeable future. O^her computers can do the J^b and we do- not wish to 
foreclose on, this decision until the last possible minute. The FDF-il 
line was selected because of our early discussions with the National j Bureau 
of Standards, which had done some related work on such a machine and [had 
oriented our thinking 'in that direction. Also, similar ^-e^ppli cations ! are 
,operateci from approximately this configuration at the National Library of 
Medicine, ^Medical College of Pennsylvania and, formerly the Universiiy of 
Delaware. -At least one larger machine, the PDP-ZO, probably can do. tW 
job with less than ,its full capacity, although at this time the BDF-il 
and PDP-20 are not software compatible. A new IBM computer, the Series 1, 
looks ■'promising also. • This was announced by IBM, after most of tfte IIDA 
design decisions had been* made, but its low price may make a switch worth " ' 
the effort. 

The- configuration assumed in the design of the softwar' is the folloi^ing: 
PDP-ll/aA, with 96,000 bytes of main memory > - ^ 

2 magnetic disk units, of 5 million byte capacity each (one largely taken 

up with operating system software). Disks are dismountable. 
A le-cliannel riniltip lexer to which will be attached one high-speed 

terminal, either CRT or printer, operating at at least. 12-:. characters 

per second and up to 15 30-cps dial-up lines. 

The programming language will be BASIC-PLUS and the operating system 7 
system RSTS/E. 

3.1.3. Software configuration 

We have already noted that the data base processor is resident in a 
computer external to IIDA. IIDA communicates with it bu". plays no role in 
its operation. The IIDA computer contains the instructional and communic'ations 
processors. These consist of the following software components; respectively^ 

!• Instructional processor . From the student point of view the IP 
consists of the tutorial courses, the exercise and the assistant mode. These 
are supported, to various degrees, by a HELP program and a series of diagnostic 
programs. The courses, exercise 'and' assistant mode, and. the HELP program, re- 
designed Ground the concept of a shell , a program which serves primarily 
as sequencing logic among a set of subroutines. Thus,, the tutorial shell' 



Tutorial shells ~ 
one for each course 



Itbcercisd shell 



HELP Sholl Assistant shell 



Dati bise 




Tutorial frames, each Jpresenting a 
unit of instruction, called^ one at 
a time by a shelly 



EsoBrciso frames, called 
individually by the 
exercise shell, * 



Help frames called 
by HICLP shell irtiicli 
is called by the 
exercise or assist- 
ant shelly 



Assistant 
frames called 
by the assist^ 
ant shell* 



Figure 3*2. Gene>^^ softi^are configuration* 
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sequences the student from one frame of training material to another, a 
frame being the unit of presentation of tutorial material. The exercise 
sequences the student from one frame to another, also. In this case, the 
frame is the unit of programming required to elicit a command from a itudent. 
HELP does its sequencing by presenting the student a series of Menus, lists 
of choices of options open to him. Any choice might rasult in another mer 
offering even more- details of the choice Just made. ^The diagnostic progrc.:u3 
are not bound to a shell. They- are called, as needed, by frames in other 
programs. The relationship of shells to subroutines is shown in Figure 3.2. 

Communications processor . This set of prograxus consists of the elements 
previously noted, In Figure 3.1: message switching, store and forward, 
logging, concentrating, billing and proctor support. 

3.1.4.. Logical processor 

Common memory and internal channels are 'two distinctive features of 
BASIC-Plus under the RSTS/E operating system. Both place constraints j, 
on and provide capabUities to the IIDA system. A short general discussion 
of these features follows. Their application in IID^ is developed further 
in this chapter, expecially Section 3.6. 

!• Common memory . Sixteen bytes in the miln memory are available for 
reading and writing by all programs in operation. The .bits of this common 
memorj area are used for rudimentary interprogram communicition. 

' 2. Internal channels . Up to a maximum of twelve data, files may be 
assigned to any legal . -temal J/O device., in the IIDA system, these channels 
are exclusively assigned to files on the disk. Files may be shared by programs. 
The same file may be read and written by different prograiis, if the protection 
code allows it. Only the program which first opens the file, however, may. 
iKctend the file by writing to it. For this reason,, communications files- 
((see Section 3.6.1.) must come in input-output 'pairs. Writing extension 
jpriyil.iges for the output file are held by the instructional program. 
Extension privllegea for the input *^ile are held by the message switching program. 

The Instructional PrOi.essor. 

The instructional proces^r comprises all the programs which teach the 
student or diagnose or monitor Vm. We have designed the IP as a set of 

shells" or programs whose funcci^iTi^ is to control or guide the sequence ' 
of other programs. There are five such shells: the tutorial, exercise, 
assistance, HELP and user control shells. The diagnostic programs, which 
make up a good bit of the programming under each shell, are discussed in 
Section 3.3. 

3.2.1. The Tutorial Shell 

_ Tutorial programs are CAI programs. Each consists of a set of farames. 
A trame is, in our usage, the program that presents a unit of tutoriiT 



^. ■ : ° ' r'-e,'"- i-iioi. t;j.cocin.s a unic or tutorial 

information, elicits a response to it from the student, and performs an 
analysis of, that response. There can be branching within- a frame if, for 
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example; a student gives an incorrect but aatici'pa^ed reply and the program 
author .simply wants him to try again, the shell then serves to determine 
the sequencing among frames. ■ ' / , 

/ ' ■ 

In a CAI program, .a shell ia quite simple. For eachirame there is a 
Ti^^hfiTi next frames and the choice is ma;i4 within the frame, c 

^Inlln t 'I frame., as a unit of/ programming, and 

branch to xt. ,If a frame leads to the decision that a' student has given too 
many unrecognizable answers, or that .the program of the frame canno? understand 
the replies the shell may branch to a generalized routine for handling 
unrecognizably replies. , iiduuxing 

, In this way, '.he CAI course/ author is- given the responsiblity = to decide 

what frame to ex-^cute next, defending on the student's reply to his question, 
but not for atiyof the programming logic necessary to invoke that- next frame. 

One advantage of this ^pproach to programming is that ^ every frame can be 
operated as a subroutine, and in any sequence.- Thus, the HELP program can : 
make use of , tutorial frames in providing explanations of commands and reviews 
of specific subjects can be assembled from existing frames. 

' c 

It is our intention to allow a student to begin his IIDA instruction 
almost anywhere and to have maximum freedori to traverse the material as he 
wishes. The shell concept, by making each .frame a separate unit of 'programming, 
assures that students can begin almost anywhere. There will be some frames 
which make no sense as starting points, but the student can be given a table 
of contents, which are effectively entry points, and allowed to choose for 
himself. 

3.2.2. The Exercise Shell 

This sh^ll has the greatest number of control options, hence decisions, 
of the five basic shells. It also makes nst of frames." but this time not 
units for which there is a right or wrong student reply, instead, there ate 
acceptable and unacceptable replies and there may be any number of acceptable' 
ones. *^ 

" - ' . '-^ 

The basic logic governing the exercise is the three-phase model of a 
search. Students will be started in Phase 1. in which they perform index 
search type commands: simple SELECTS and EXPANDS. 

c^ A ^"u "^^i three different levels of exercise shell, one to take a 
student through a "canned" search in which he is told what to search for and , 
how. and is given the experience of doing it. The second permits him to do ^ 
a search of^his own. using a limited subset of the command language. This ' 
will be similar to the exercise ia the IIDA model. The third shell works 
with the full language. In all cases, the exercise shells are' built around' 
thfe concept of a search phases, and they ex^iect the student to begin with 
phase 1 and w6rk through to phase 3. ' ' - 

T"^^' ^ ^^^^ exercise, is . command. In most cases, 

the student may enter any number of commands, so long as they are valid. 
An exceas of invalid' commands may be treated essentially as a sequenc4 of 
unrecognizable replies in the CAI mode. Progression from frame To TvLl 
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Frame No. Frame topic 
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Figure Shell structure for the 'limited language subse% version 

of^the exercise. 
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Figure 3#U# Shell structure for assistance mode* 
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-requires completing a use of each of the commands defined for that exercise 
level. After the student reaches and completes phase 3, he may retuni to 
any earlier phase of his choosing. ' 

A student may make use of what amount to phases 4 and 5 if he usas an 
administptive command or" a request for information or control. .Thus, 
HELP is ja. phase 5 command, valid at any time. BEGIN is a phase 4 coipmand, ' ' 
valid' in the exercise mode only before commencing phase 1. 

-The shell for the exercise in the model consisted of 11" frames , and 
probably not much more will be .required in the full implementation of any 
level of exercise shell. Each frame elicit^ a series of commands' of a given 
class from the student. Analysis of the commands takes place within the 
frame. Transition from frame to frame • takes -place upon completicj of all 
phase requirements, with the understanding that transition to phases 4 or 
/rn^^T^^^® at .any time, or is dependent upon the specific command 

CLOGOFF is not -valid as a first command, etc.) 

■ Figure -3.3 illustrates .'the basic set of frames to be used in an exerci/e 
and tna connecting logic needed. 

3.2.3, The Assistance Shell . . 

This shell although it drives the most important of the instructional 

! simplest structure. There is no prescribed or&er of commands 

Jor ^"P' '^he logic of the DBF: that. 

or tr^ECr Efi I? '° having defined any sets, 

°o E6 dffa^J?"^ ^^^"1?? ^^^^"8 ^ '^^^ corresponding 

to E6. The diagnostic programs will detect and comment upon such misuses. 

^stL ^ssistanc\ shell will accept any input, have it parsed and results 
^If I A P^'^^°'^"" ^story files. The performance analysis routine 
examines student performance, .looking for indications of unacceptable or 
ZlJfr'i''^ behaviour. PAR serves as a sort of after the fact frame sequencer, ' 
consistent or meaningful in light of what happened previously. , 

r^mr. ^^^J^^'^f' ""^^ ^ssistance' shell calls the frame walch elicits a 
command. That frame calls the parser as a subroutine and the parser pasP- • 
the command, if valid, to the CP, for transmittal to the ^P. When the DBF 
response is received, the assistance shell calls a frame which calls 
updating program that posts results of the command to the performance 
tiies. Then, the shell calls the performance analysis routine.' This 
sequence of events is illustrated iu Figure 3.4. 

3.2.4. The /HELP Shell ' , ^ ' 

/HELP is a legitimate command at any time, for an IIDA user. There are 
^ny different ^forms of help available. The /HELP shell is invoked from 
another she.l whenever the student calls for this facility. The/HELP shell 
^iZr , of the programs needed to pr^esent to the student information about 
what is available, and to ask him what he wants.' Providing the help then 

Figurel^f" ' ''^^ °' ^^-'^^-^ is UlustJited 
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Figure 3. 5^, /HELP shell , shov Ing options of fered 
at various levels. 
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The basic shell presents to the st\dent a "Tree" of menus » i.e. a series 
of lists of features available. Each tl^ he selects one, a more detailed 
list is provided, until he has finally made an action decision,' i.e. one 
calling for some substantive information to\be provided, a diagnosis run 
or a change in sequence of the program. 

3.2.5. User Control Shell 

'ks described in Section 2.4 there is a set df Wtrols by which the^ 
student can to some extent modi'fy the .sequence 6f\lDA operations. These 
commands are directed solely at IIDA, not the data base processor. T 
assure that they are not mistaken for DBP conmands , \ach is preceded by the ' 
character / . For each command there is a program, V set of frames, 
designed to, implement the command. All that is requir^ of a user control ''■ 
3hell is that upon recognition' of the issuance of a us^ control command, 
program control be transferred to the appropriate frame. \ 

. The .user control shell, therefore, has very little toMo. A shell is ' 
cheated in anticipation of future expansion in the number aiid complexity of 
user control functions. umpxexity oi 

3.3 Diagnostic Programs 

Diagnostic programs are invoked, in general, by program frames wltMn .,n^' 
of the instructional programs. Their purpose is 'to^anaJ^rbotHhe^ost "''^ 
recent student input and the entire pattern of this and previous , inputs: 
^alysis is done at three levels: diagnosis of the current student Lp^t, 
micro-analysis and performance analysis. (See Section 2.6.3). 

Analysis of the current input is done by two programs: the first looks 
for control commands, those beginnihg with the cha?acLr "/". If an Lput 
is not one of these, it is asstimed to be a DIALOG command and is passed to 

sLtJr? ^'ir^nti'^' '^'^'^^^ °" procedural and syntactic coTr'tnLs ' 
(Section 3.3.1) and interacts with'the student if the command is in any 

Ttuden-r''?''- ^"'i?"'^^ '""^ '"P"'^ °' an ^acceptable SSlOG comLndl the 
its r^;'.^'" ^'"'"'^ ^''^^^ "P^^*^.^*^ ^° «how the new command and 

DrLr!^^ . """"^ microai^lysis. The microanalySs 

orSS^f°^"Jj^ data ,ext,racted- from the student's coumanfto produce 

tSe ^A^ ^^L'"" ^"'"^'^ performance history for use by 

the PAR. These programs create.new data elements. They do not make decisions. 

stratSc'nSif%'''^ '"'^'^^ °" performance history^ files for signs of'' 

When the preliminary threshold checks indicate xh'^ existence of nv«i,i 
and occasionally eVen without such indications! more detan^^ n^^ P^^lem,. 
are made of student performance. ations, more detailed pattern analyses 

The command parser is described in Sectibn: 3 3 1 The a 
performance data base is described in Section S.S.'s. ! 
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3*3. 1. The Command Parser 

The Parser will verify the syntactic and procedural acceptability of ' ' 
commands entered by students, identify the nature vf errors whenever possible, 
and ^provide information to the student or a calling program on the nature of 
any errors detected. In its most common use the parser will converse with - 
the student about a command error and try to elicit from him a correct form* 

The parser will detect errors of syntax, i.e. commands which do not 
follow the rules of a construction of the search language, and errors of 
procedure, i.e. attempts to use commands which, for tutorial purposes, are 
not . acceptable at the time, even if correct in syntax.. The parser is a 
subroutine.^ The program that calls it will provide it with Information on 
what commands are procedurally acceptable at the time of calling (including 
the possiblity that all commands are acceptable) and Will specify the level 
of analysis desired on the command's argument. 'When used by a tutorial 
program, a limited form of analysis is needed because the tutoring program 
can, normally make an exact match comparison on the argument, while in assistance 
mode, there is no way to anticipate what the argument should havet-been. * 
Therefore, a tutoring program will ask for a minimal amount of argument 
analysis but the assistance program will ask for the maximum amount. The 
levels are: . 

1. No analysis. The parser will return to the calling program the text 
of the argimient, for its use in making comparison. 

2. Blank extraction. The parser will remove all blanks from an argument 
and return to the calling program: the original text, the blank-extracted 
text, and the number of blanks extracted. A calling ptogram would likely 
find the blankless. form of the argument easier to test against a standard. 

3. The parser will completely parse the argument and if no error is 
found post the components of the argument to the various performance history 
tables. If an error is found, the error text is posted to the error file 
and an error indicator is turned on. v 

Overall Logic Description. All DIALOG commands begin with a verb, 
an abbreviation, character representing a verb, or a period followed by a 
verb. Most then have an argument, such as the descriptor following a SELECT 
command. But noj all command verbs have an argument, e.g. LOGOFF 
The Parser identifies the verb first and then the argument.' If a*valid 
verb cannot be identified, no attempt is made to parse the argument." When* 
an error is discovered, it is often not possible to be certain that what 
the parser detects as an error was the actual error. See the example in 
Section 2.6. of how an error <pn the part of the student may lead to an 
ambiguous situation. M ' ^ 

In general, there are two principles which guided the design of this ' 
program: First, it is not possible to resolve all ambiguity in interpreting a 
student s input; however detailed the analysis, the program cannot know what ^ 
was in the student^s mind nor can it know how many mistakes may have been 
made in a single input. Second, I^DA is an on-line system that must provide 
rapid response to the user. It cannot take too much time to execute any 
.of its many programs. Hence, parsing is done on this basis: first a 
command is recognized and if one is. found trffe argument is parsed. In parsing ' 
the argument, a quick decision is made as to which form £ argument this is 
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. (several commands, notably SELECT, have more than one form of argument) 
and any errors or^ inconsistencies are interpreted iii the light of that 
assumption. In the worst case, this could be somewhat confusing to a student, 
bat it cannot lead to error. In other words, it could lead to the Parser 
telling a student tliere is an error in his type a argument, while he actually 
intended a type b argument, but made enough errors to make it look like type a. 
But the parser will always tell the student what assumption it made and 
erroneous assumptions on its part catt be quickly overriden by the student. 
We do not attempt to use the argument to help identify the command entered, 
nor do we use any evidence other than that involved in the first cursory 
test to determine vhat form of argument was attempted. Frequent and rapid 
com?;unication with the student can ^overcome these lacks and the saving in " 
delay response can be great. 

Identifying the verb. The logic of verb recognition must follow the 
construction of the language and the rules used "by DIALOG in it-j parsing 
For example, if DIALOG finds that the first n characters- of a command constitute 
a legal verb, it assumes that to^be the verb even though thtire may be other 
interpretations of the command, as in SELECTION. This command will be 
interpreted as SELECT ION, rather than S ELECTION, both of which are correct 
forms • 

. The IIDA parser examines the first character of the string of characters ' 
constituting the student input;. • Each initial character delimits the number 
of complete command name? that need be searched for, and if the initial 
character is not a valid first character (i.e. not the initial character 
ot a valid command) the parse can end there. For, each initial character, 
the commands beginning with that letter are scanned for. If the full verb is 
H?^:^°"''i',i=*'^" the character is a coknand abbreviation, • the parser assumes 
that is what was intended. If the first character was not, itself, a valid 
abbreviation and no full verb can be, found, then there is no valid verb and 
the parsing halts with an error, indication^ 

In scanning for either verb recognition the. parser is guided by input 
from its calling program. Procedural restrictions are placed upon commands 
by the calling- program telling the parser what commands are acceptable at this 
Point in a search. A- student may, then, enter a syntactically 

^^il''°T"'^ ''^^f unacceptable because it was out of phase, or for some other^ 
reason. No procedural restrictions may be placed upon the argument through ^ 
this means — only the command verb. Argument checking is done by the callinz 
program. * 

ft 

^- Conmand Recognizer. Specifically, the following steps are taken 
in recognlzihg the comDmand verb and parsing the argument: 

1. Scan the entire string (as received from the student, consisting 
of the command and argument) . If there are blanks preceding the first non- 
blank character, shift the entire string left the number of blanks that occur. 

- It there are blanks following the last non-blank character (excluding the 
carriage return or end of message indicator), delete them. Make a copy of 
the remaining string that has had all internal blanks removed possible for 
use^by the tutorial programs. ^ 

2. Scan the. string and record the position of all occurrences of the 
following characters: »(.)-,?.. 

ERJC - 0 . 



Cbniman(i Verbs- 



Abbreviations 



Symbols 



BEGET 

CQMBDE 

DISPIAY 

DISPIAT SETS 

EHD • 

SND/SAVE 

aiD/SDI 

EXPAIID 

.EXPIADI 

KEEP 

LTT'ITT' 

IDGQFF 

PAGE 

millT 

SEIECT 

TYPE 

.SXECUTE ^ 
.FIEE 

JIECALL 
JIEEEASE 



none 


.1 


C 


$ 


D ' " 




DS 


Q 


none 


m 


< 

none 


•/SAVE 


none 


•/SDI 


E 


II 


none 


• 


K 






) 


XAIIi 




nono^ 


none 


P 


0 


m 


& 


s 


# 


T 


f 


none 


none 


II 


II 


VI 


II 


II 


II 


II 


II 



(not used in HDA) 



< II 
It 
II 
II 
II 
II 
II 



(hot used in HDA)^ 



II 
II 



(not used in IIDA) 



It 
II 
II 
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NOTE:^ The notation not used in IIDA refers to "the single- 
character abbrcfviation, not the. command or its abbreviation. 



Figure 3^6 Alphabetic list of DI&LOG command verbs, abbreviations 
and symbols • . v . * ' 
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3. Perform command recognition. The result of this subroutine is the ' 
return of: an indicator showing whether or not the command\erb was valid, 

an indicator showing whether or not the verb was immediately followed by 
a blank, the command or its abbreviation or symbol,' whichever was used b/* 
the student. 

4. Perform an argument scan. There will be a different scan program for 
each comnand yerb. If a command that should not have an argument has one, 

the student will be informed that IIDA is assuming the command entered 

and asked if the assumption is correct. He has' the chance either to cancel ^ * 

cuw the needless argument or to change to command. In general, they will: 

a. If the command -i* valid ^ tell the student the assumed source of 
thdVg^ror, post the nature of the error to the error history, count the 
number of times an error has occured on this command, and elicit a 

^ new input from the student. 

b. If the command verb is valid but no blank follows it, tell' the 

^ ^ student that the command is being assumed and offer the opportunity for him 
to reenter the command if the assumption is wrong. Suggest he. use a blank 
separator in the future. 

If the argument is blank or null^ i.e. missing, reject it, even 
though DIALOG will accept it. Allow him to override this rejection. 

d. If the command verb is valid but the argument -is not, inform the 
student of the nature of the error, post the error to the error history 
file and elicit a new command, counting the number of times this is done. 

e. If more than a to-be-determined numb.er of tries is necessary, 
invoke the PAR. 

f. If * command , verb and argument are valid, post the elements to 
the appropriate history files, set an indicator to show valid command, • 
and. exit from the parser. ' 
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show: 



Program Losic. To illustrate the program logic ^ the following figures 



Figure 3.6 the list of DIALOG command verbs, their abbreviations and symbols. 

Figure 3.7 the recognition logic for the first character of the command 
string.* 

Figure 3.8 the illustrative logic for one particular initial character, 
E, which could lead to any of several command verbs. 

^ .p,^?;?"''^ 9 the Backus-Naur formal representation of the argument of ' 

QTrTPTT ^^^^^ ^ formal expression of the ways in which a valid 

bhUiLi argument can cr.cur. 

Figure 3. Da flow chart of the parsing logic for the SELECT argument 

In Figure 3.6 the complete list of command verbs i5 shown. Some of these 
Jwo wordfT^"' K? f"^ ^° In- several cases, it is arbitrary whether 

two words are combined as a verb | or one word is the argument of another, such 
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Figure 3.7. Cec^ton table for anaO^Tsia of first character of 
coininand string. 
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01 - 07 • EXPIAH'I i T' .' IT 

01 - 03 • EMD 'I' H 

01 - c8 - aiD/sAYs ; i ■ r- n 

01 - 07 - EIID/SDI ■ ! ■ T 'll 
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00:-2I/UID/«i 3© i "X: 

OOiriAlD - SD/iDAVE i ix! 
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Figure 3.8# Decision table for reconiition of comraand verb on 
<2fiG5*assuint*>:'on that first oharayter (d) » »EU If no full verb 
naoe is found, the comnand is assumed to be SICP.'UEI, entered in 
abbreviated fonu.as E<» 
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as DISPLAY SETS, which defined by Lockheed as a separate command wirh.no 
argument, but could be interpreted as a DISPLAY command with the argument SETS. 
DIALOG recognizes the (usually) single-character symbo-,, a hold-over from 
the early RECON system.. Because these are not mnemonic we do not plan to teach 
them, and will not recognize them except for the ? (EXPL.'^), It cun be seen 
that the first character often, but not always, identifier; the command verb. 
In all cases, however, test-ing of the first character reduces the number ' 
of further tests that naed. to^be made. 

Figur.<i 3.7 is in the form of a decision t^bxe. It shows what decision 
is m^ie from the first character. The decision is always one to branch to 
a next level of testing unless none of the accepted first characters occurs. 
In that case, the command is immediately declared invalid. 

Figure 3.8 is a decision table illustrating the logic for secondary 
testing after the first character has been found to be E. Such a command 
could be any of the five listed. If none 'of them, it is assumed that t'ue 
command was abbreviated by the 2 and thus treated as an EXPAND command. 

^- Argximent Parking jach tjrpe argument for each, type comand must be 
formally defined as to acceptable syntax. Form includes both syntactic 
structure and in some cases content. For example while nearly- any string 
of characters is acceptable as a descriptor, the number of descriptor suffixes 
or tags is quite limited. An illegal suffix or tag invalidates the argument, 
as does such an error as omitting a / in the argument of a PRINT command. 
For example, PRINT 3 2/1-4 is invalid because there is no separator between 
the 3 which identifies a set number and the 2 which identifies a format. 
While separation by a space rather than a / seems acceptable, without 'the 
spac^;: there is no way tc know whether tht student intended PRINT 3/2/1-4 
or PRINT 32/7/1-4, etc. Rather than guess or dwell overly long on wf. mlgh- 
have gone wrong, as soon as an error is detected, the command ^3 sent back to 
the student. 

Figure 3.9 shows the formal deflnitir- of the SELECT argtnnents. We 
have chosen this for illustration because it is niore complex than -moat. 
There are the following types of arguments for a SELECT: 
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EXSTR (Expand string) which denotes uae or more line numbers (EXEL- - 
Expand element) from an expand display. Tb^se are of the form Enn or 
Rnn and may be connected by commas or hyphens. 

FREE (Free text) which connotes, the ^form used when searching the text of a 
field for co-occurrences of terms, this configuration always has a set 
of parentheses embedded In it and within them must be any of a limited 
number of link codes. 

SUFF (descrip^r with a suffix) which has the form: descriptor/SU 
where SU can be any of a limited number of suffix, codes. This form of 
descriptor may be used at the end of a Free argument. 

PREF (Prefix) which lias the form: TG-descriptor, The must be in 
position three and the TG may be any of a limited number of tags. 

STRING (any string of characters not one of the above) A string may 
tennlnate in ?, implying it id truncated (TRUN) . 
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U1 < dftfj > «^j9ffin^ > I </>ref > « <«fn'n5 > j < de^c >/5ttff I 

<fr«e> l<ex sk> 
[Si <5^nj>::« dejc I itit ? 
iHl <prt(> At;/... 

C51 <5off> j: • tS !,..{<' '^f>,}<SU^>Tl|Dt'(l0|C5(Aa^ • 
M <frt« > • {dcSfi. (•4|iV , )] dcsc i <fre€>/<9uH> 
C7I <link>::« wIlIs <(?t^> Wl— 
Cftl <ex5fr>:;« <exel> ( f<€X€i>'^ } <exel> 

R<ik>-«<i!2l> 

<in}> - Moie thif e«rftin limits Appiy, 
t^. in £<|of7.<iiii» mpy not 
ttCfcd Urjesf number in DtP^NP 

int, 3'inijt , 

Tcrmiri^i Sjfmbol5 

tdftidi And oiHer spbob are likrtis. 

Figure 3.9« Backus-Naur representation of theVjmtax of the 
SELECT coramar.d. • \^ 

' ' 57 \ 




IIDA treats an argument of the FREE type or EXSTR, if it uses more than 
one line number, as a phase two comnand, one concerned with combining 
terms not just defining or searching our Individual terms. 

Figure 3.10 shows the overall flow chart for the argument analysis fol- 
lowing a determination that the command verb has been SELECT or S. 

First, test for the EXEL or EXSTR type argument. EZEL and EXSTR^ of 
course require that one or more EXPAND commands have been given. If at least 
one valid EXEL is found, then assume this is the form Intended, treat all 
subsequent errors in the argument as deviations from this form. 

^''^ " argument is not EXEL or EXSTR and if there 

ireat lllT.T. P^^^^^I^^««« with non--blank characters on either side, 

treat this as a FREE and any errors as deviations from FREE. 

Next, test for SUFf . If/ occurs in the third position from the end of 
the string, check for valid suffixes. If it occurs in, any other position, 
treat as an error in trying to form a valid SUFF argument. 

Next, test for PREF. If the character « occurs in position three, check 
for valid tag. If « occurs in some other position, treat it as an error in 
trying to form a PREF argument. 

■* 

Next', test for a truncated descriptor. If the character ? occurs anywhere 
except the terminal position (if first, it would have been deemed a verb) 
treat as a possible error. 

Any other character string is accepted as a valid descriptor, except 
an all-blank argument. 

3.3,2. Performance Data Recording 

Sources of the da,ta written to the student performance file include : 
(1) the command parser which evaluates student entries, (2) response parser 
programs which capture significant elements from data base response? and 
(3) expansion routines which "explode" codified data into more useful 
form and (4) output of microanalysis and "PAR programs. 

The command parser and its resulting output were discussed in Section 3.3.1. 

The response parser is ai set of programs which are called to examine 
a data base processor response to a student command. The response error 
detection routine is called automatically. Its purpose is to Identify error 
messages or changes of state (e.g. the system going down) from the data base 
machine. 

» * «■ - 

The other response parsing routines are invoked to extract data from 
responses to a specified group of conncands.' The SELECT,- COMBINE and LIMIT 
command responses provide information needed to update history files The 
EXPAND and RELATE commands elicit tables referenced by various other 'programs. ' 

The expansion routine's are called primarily to translate truncated 
and E- and /R- series arguments into more meaningful formats for analysis 
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Figure 3. 10 (Part 1) 
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'NOTES: (1) The parse subroutines for specific types 
of arguments begin with the assumption that the 
argument is of the stated type* The^ detect errors 
and post either the coirect data elements or error 
data* (2) the « Inform student » subroutines tell ^tJie 
student the nature of an error, then go to a sub- 
routine that counts the" number of re-tires and 
elicits another command* 
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Figure 3#10 Schematic of argument parsing program, shovring quick 
decision logic* 
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ptHTposes, The expansion routines are called after the user has entered a 
SELECT coifflnand with a truncated or E-/R- Series argument. 4 truncated term 
automatirally generates and sends EXPAND commands to the data base* ^ The 
descriptors returned with jche designated root word will be. stored in a 
descriptor array for use by the calling program. 

If the argument of a SELECT is coded En, the program addresses the n-th 
position descriptor of the EXPAND table. If the argument is a range of ' 
reference numbers, e,g. E1-E6, the first . through sixth descriptors are 
^transferred to the descriptor variable array. When the argument is coded 
Rn or Rn-Rrrhtt, the RELATE table is consulted and the related terms moved to 
the descriptor variable or array. 

Tiie net effect of calling the expand 'routine every time a SELECT command 
with expandable arguments is entered is the maintenance of currency in the 
set history file. If a set represents a combination of multiple descriptors, 
the terms will already exist in expanded format. The last generation of 
updates will include all "expanded" data thus eliminating redundant queries 
to the set history files. 

If expansion of truncated or E-/ft-3«ries data results in lengthy lists of 
terns, an abbreviation code is used indicating that the succeeding data 
elements represent: the term which has been expanded or related, R or E, 
first and last term included. Length of descriptors will be interleaved. 
Terms and tables can in this way be regenerated if desired. 

The sequence iii' which data is recorded is important, .especially to the 
restart and recovery procedures. The command parser posts data about valid 
commands and errors made as it discovers them. When a valid command is 
achieved, it is sent to the DBF. The response from the DBF goes to the 
response parser which posts it to the performance history files. The expansion 
routines are service programs used to support various other diagnostic programs. 
Finally, performance analysis programs are run and they, too, may add data to' 
the performance history files. The history updating cycle is complete only 
when FAR has finished its work. Indeed, a bit is set in the command history 
file when FAR has completed is final work on a given command to show- chat all 
files have been updated with the elements and consequences of that command. 
Any command for which all processing is not complete must be repeated in 
recovering from a, computer hardware failure (See section 3.8). 

3.3.3. Data analysis programs 

These fere a group of diagnostic programs that are primarily used to 
support other diagnostic programs. They consist of a program for phase 
analysis, clustering, expansion of commands and responses, similar term 
matching, and response parsing. 

1. Stage analysis. The purpose of this program is to classify the commands 
of a search into groups of commands which cohere as stages. We have previously 
defined search phases . A stage is an instance of a phase. For "example, a 
search might proceed from phase 1 to phase 2 back to phase 1. We would call each 
of these a stage and number them 1, ? and 3. Stage analysis data is added 
to the command history file and is u.ed by the performance analysis routine. 



Stages are primarily, recognized by the predominance of one of the five 
phases of searching: index search, logic formation, record display, procedural 

^t^tT^^''^"' '^^^^ ?° ^ "^^^ P^^^^ «hich no phase predominates. 

Pragmatic rules are applied to distinguish borderUne cases. These niles 
will be discussed below in terms of three passes through the search. 

Each command is assigned one of five phase codes depending upon the ' 
pnase to which the command ideally belongs.. Groups of commands, referred ^ 
to as strings,' will become stages, either in their own right or in 
combination with other contiguous strings. 

ru..J^ the first pass of the analysis, "nuclear strings" are identified. 

S the^^L' H °^ ^° °^ ""^^ contiguous Commands 

of the same phase, or (2) any tx^o or more contiguous commands «Ech of a 
dltterent phnse. The former are pure strings dominated by the given 
phase; the latter are mixed strings in which no phase dominates. Limiting ' 
^rMr^!^ ! ^'i"^' pair*'. of commands in phe sai.e phase is a somewhat 

revLln^/?? The choice of this and other constants are subject to 

revision following . experimentation. 

On the second pass, "string capture" may take place. This involves 
merging short intervening strings with longer bounding strings. The fdllowinV 
K ^i^i^'' intervening string must be shorter than the boundinrstrings- 

the bouhding strings on each side must' be dominated byX sle pSse- 
and the intervening string may not exceed two commands. This process'is 

th!n f ^ f C^P«=^"« become so extended that less 

than 70% of the commands belong strictly to the dominant phase. Both 

2d s'^bj^c^to'r"!' strings,.ust be nuclear and the 70Z rule are arbitrary 

By now strings haVe become stages. On the third pass, each sta«e is 
analyzed. for its relationship to its contiguous stages! ex; e Sally the one 
wh«e tSf Rf-tionships may be of th^e -ypes: 1 g^nsitionkl 

2? excSsio^T t^^" ^u'^'t "'^"'^ "^8es whose do minant' phases differ; ^ 
IL T i r ^' where the stlge falls between tvo other stages which have 
the same dominant phase; and 3) terminal where the stage is at one or 
the other ends of the search. ~ 

of d^SS/''^*^?.r^^''^°"^^^P ^'^^ predecessor is also evaluated in terns ' 
A. ZTr. t°l'f distance. Direction is positive for forward movement ±1 
for tltZ i^"^^ '^^'''^ '° ^°8ic formation to record display) and negative 
aZ,^ "^-T^"' ^^^'^^^^ i« the dif f erence be^Sn the 

dominant phase of the present and that of the previous stage. In ge^eSl 
it is a measure of the "steadiness of progress" in the search. ^ ' 

:,n«i^I?"^' ^^""^ ^^"^^^ parameters are generated in the stage 

analysis program: dominant phase code, relationship code, -and directed 
■ n ^'^r "^8e. Stages identified by this anSysis arl 

a?e Id Z^'.""™^"' ^'''"'^ ''^P^^y- ^« associated parameters 

are used to direct an interactive discussion of strategy with'the student. 

2. Clusterinf^ pro|^rain. The purpose of clustering orozram la tn Ho^or™■f«o 
When the student has changed tack in his search. i!e!'s?opperpursiing 
variations on one particular combination ^of descriptors and begun^rs'uing 
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sets based on a different collection of descriptors. The purpose in doing 
this is to detect the extremes of excessive dwelling on one combination without 
detectable progress toward an acceptable final set, or frequent change of 
direction without taking the time to see if the initial logical combination 
can be Improved upon. Clustering in this context is not quite the same as it 
is in such contexts as docximenc classification* We are not looking for 
elements (commands) that are similar to each other; we are looking for points 

at wtiich a sharp break is made with work immediately preceding. 

> 'I 

The following terms are used in the logic description: 

Ci, Ck :: » The sequence of COMBINE commands (or multi-term 

SELECTS in order of entr^^by student. There 
may have been intervening non-COMBINE or SELECT command 

\ Number of index search or print (or ether 

non-*COMBINE/ SELECT) commands preceeding 
command Cj^, 

\ Number of descriptors used in COMBINE 

(henceforth used to include the multi- 
descriptor SELECT) command Cj^. 

Mi J Number of descriptors in common to COM- i 

BINE command C^ and Cj. 

^^k -5" Number of COMBINES in succession that 

have indeterminate (see definition below) 
similarity . to predecessor. 

<» 

Si,j Measure of similarity between command 

C^ and C4. 

Si,j - l72(Mij/Di + Mi j/Dj) 

- M/2(l/D; ± 1/D;) 
Si,j is the mean of the number of matching < 
terms in the two commands divided by the 
number of terms in each of the two com- 
mands. 

^nx^n command in cluster m. 

hiiyl The number of commauds in cluster m that 

match (S^ ^IT^) 

command d 

The number of links between cluster m and 
cluster p. The number of links is 

" iep 

Threshold value of ^ such that If 
Sij>Ti then q is similar to C j . 

^2 Threshold value of S^j such that, if 

^i not similar to . 
NOTi:, If T22Sij<Ti then the similarity 
between C^ ,and (Z is indPtPrm^na^o 
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is indeterminate. 
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Ta Threshold value of Nfc such that if 

%-T3.then if S^^i^^ is Indeterminate, 
S'± and C^^^ are -considered not in the 
same cluster. * 

The procedure is as follows: ^ ^ 

1. Whenever a new COMBINE or SELECT with maleiple terms is 
received from the student, and is syntactically' valid, store 
it in list as Cj^. 

2. Compute % and-D^ if i>l (use actual descriptors, not sat 
.numbers). 

3. Compute S^^^^^ if i>l. 

4. If S±^±^^>T^ then'Ci is in the same cluster as C^^^. Add 
Ci to cluster Gta,n» increase value of n by 1. 

If Si^i_^<T2 then start a new cluster, i.'e., m « mfl, 
reset n to 1. 

If T2iSi,i-i<Ti and N^IiTa then start a new cluster, i.e., 
increase m by 1 and reset n to 1. 
ELSE R± « Ri-x+l 

5. Compute L^^± and K^^p for all clusters from 1 to m-1. 

The data available from this procedure is; 
number of new clusters formed 
number of COMBINE commands given 
average number of commands per cluster 

length of string of successive indeterminate similarity to 
previous eommand 

last set number created in each string 

array of cluster-cluster similarity measures 

indicator: was a new cluster bjegun with the most recent 
commar.d 

^' E^cpanalon routines. 'One of the advantages of a language such 
as used with DIALOG and other search services is that condensed notations 
can sometimes be used to avoid having to enter lengthy, often repetitive 
statements. For example, following a DIALOG' EXPAND command another EX- 

8^^"' reference to a term on the first 

^TSS «f ^ number. The ntixt command, could be, for example, 
EXPAND. R6 or SELECT E12.E14. One of the functions of tie expSn ;ou- 
tines is to trsnalate such usage back into original descriptor form, for 
storage and analysis purposes and for later use in displaying history ^ 
£r 13 ^"'^^ P'^og'^ams are planned. One translates 

t^ r nw^? °r5^^f f Illustrated above, one translates set numbers into' 
• J^lrti '^V^t^'^''^ descriptors, and one retains information about 
^**at terms might have been seen on an EXPAND display. " 

^=-^!^^Jt-SSm^' Usage such as E6, R9 or E2-E4 are per- 
^Itl l u °f SELECT and EXPAND commands provided 

requisite number of preceding EXPANDS. Tills 
expander converts each term so referenced back into 'the original 
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descriptor the line nusnber represents and stores this information 
for use with any program needing descriptor information. 

2* COMBINE Expander > This program converts set numbers used with 
a COMBINE, into descriptors or combinations of descriptors Only 
SELECT may •create a set using descriptors, rather than set numbers 
in the definition. Thereafter, COMBINE commands build upon pre- 
viously defined sets, as in the sequence 

SELECT A (Set' 1) 
SELECT B (Set 2) 
SELECT C (Set 3) 
COMBINE 1 AND 2 (Set 4) 
COMBINE 4 OR 3 (Set 5) 

In analyzing cluster formation and in displaying set history, 
the numeric-arguments of the COMBINE commands will be converted 
Into descripfora. In the illustrated case, the results would be: 

Set 1: A 

Set 2: 3 

Set 3: C 

Set 4: A AND B (rather than 1 and 2) 

Set 5: A AND B OR C 

3. Expand Index Program ^ The function of this program is 
to record the first and last descriptor of every EXPAND table 
sent by the data base processor in* response to a student com- 
mand* Input required by this program is generated by a sub- 
routine which when called extracts fields of the EXPAND table. 
These field values will he stored In subscripted variables 
indexed from 1 to ni (n « number of lines sent by the DBP minus 
header lines). c " • 

The program examines the descriptor field arr^y and isolates 
the first and u-th position terms. The length of each of these 
descriptors is computed using the Basic-Plus 'LEN function which 
returns the position of the rightmost non-blank character. 

*' ■ 

The line- to be. written to the student history file is a 
string concatenating the following data elements: 

EXPANDED \ LENGTH LENGTH ^ 

INDEX - \DESCRIPTOR I DESCRIPTOR 11 DESCRIPTOR I IdESCRIPTOR 
-ENTRY \ A B A B 

(Fixed Fields) (Variable Fields) 

'.This entry is written to the next block number and address 
recorded in the general series data on the seventh block. (See 
3».5 . 7 . ) \ 

General series data are updated by incrementing the total 
number .of descriptor pairs and computing the next entry block 
and address. The directory references to pairs are also re- 
ordered to correspond with the alphabetical arrangement of 
first "descriptors. 

u (J ■ ' 



4. WorS similarity program . This- program is used when testing 
to see if a .zero set based on a single descriptor used a term not 
seen by the student on an EZPAND display. This requires that an" 
EXPAND <descriptor) be issued by IIDA 'and thd results analyzed by 
this program, but not transmitted to th^ student. The descriptor 
will be the argument of the SELECT that led to a zero set. It will 
appear in position E6 of the display. All other words on that list , 
will be compared with the base word to compute, for each, a measure** 
of closeness of spelling. For each word compared with the base • 
word the nuaber of characters in common will be counted. A high 
proportion of letters in common does not, of course, imply semantic 
Closeness. The table of measures will be sent to the. student for 
-his information and decision, 

5- Response parser. The response parser examines and breaks • 
down messages received from the DBP in response to valid coianands. 
It must first check for error or change of state messages. Failing 
to find one of these, it knows, from the command issued, what form 
of response is expected. It should verify this, which, can normally 
be done by checking only the first few characters of a response. 
If the correct form of message was received, the parser breaks the 
message into its constituent elements and posts the relevant ones 
to the various history tables. This program is clearly specif ic 
to the data base system used and is subject to change with any changes 
they may make in format. 

'3.3.4 Performance analysis 

The performance analysis routine (PAR) monitors five aspects 
of the student's search. These five aspects are: (1) zero set 
formation, (2) phase and cluster analysis, (3) /HELP and time use, 
(4) error analysis, and (5) extraneous and redundant commands. These 
five areas are monitored on two levels. The preliminaiTr analysis 
portion of the PAR is designed to catch problems in the search by 
comparing some rather simple statistics calculated in each area with 
threshold values. When the threshold values are exceeded an in- 
depth analysis of the area is run. There are appropriate in-depth analy- 
ses for each of the fi^^e .preliminary analyses. The Irt^epth analysis 
performs calculations designed to enable the program to assess why 
the threshold was crossed. The reasons for crossing the thresholds 
are not unique. Each threshold can be exceeded for one of a number 
of related reasons. The objective of the in-depth analysis is to 
determine which of the possible reasons is the most plausible. There is 
always the possibility that none of the planned reasons appear causal. 

Preliminary analysis is carried out after the response parser 
and student data base update are completed. As protection against 
loss of information in the event of a system failure, a system of flags 
is used. A master flag is set to show "processing" status when the 
preliminary analysis is begun. As each portion of the preliminary 
analysia is completed, a minor flag is set. When the preliminary 
analysis is complete, including any in-depth analyses that may have 
been called, the master flag is changed to "waiting" status. Con- 
trol then passes back to the shell for the decision on successor 
process. 
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1. Preliminary analysis > The preliminary analysis is designed 
to operate in a ainiimnn of time. The threshold checks made are listed 
below. Actual threshold values will be determined experimentally. 

TCI. /HELP use 

Is the number of /HELP calls Tl% or more of the total 
number o£ commands entered? 

\ Is the ambunt of time spent .using the /HELP procedure 
more than T2Z of the total time of the search? 

, TC2. Error analysis 

Have more than T3Z of the commands of any one t^^e 
been in error? 

Have more than T4Z of the last T5 commands been in 
error? 

, - 103. Zero set formation ^ 

If the most recent comnand formed a zero set, have more 
than T6Z of the last T7 s^-forming commands formed zero seta? 
TC4. Clustering threshold 

If this command is a COMBINE-llka command, la it in 
the same cluster as the previous COMBINE-like command? 

If yes, does the number of consecutive commands in 
this cluster exceed T8? 

if no, is the average number of commands per cluster 
less than T9? 

TC5. Phase checking 

Is this command in the same phase as the last command? 

If yes, is the total number of consecutive commands in 
this phase greater than TIO? 

If no, is the number of phase changes in the last Til - 
commands greater than'TlZ? 

TC6. Extraneous commands 

Is this command a duplicate of a previous command? 

If yis, is it a command for which repetition la meaning- 

ful? 

Is the total number of duplicate commands in this 
search greater than T13? 

TC7. Time use 

Was the time between the response to the previous com- 
mand and the entry of this greater tlian Ti4 seconds? 

Is the total number of commands in. this search whose 
response time is greater than T14 seconds greater than T15? 

6V 
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2. Control sequence > The order in which the thresholds are 
checked isfshown in^Figure 3.11. In the following discussion, the thresh- 
old checks are referred to as TCI or TC2, etc. These numbers corre- 
spond to the order in which the threshold checks were described in 
the previous section. 

The type of command being .checked determines which threshold 
check is* used fi,rst. The command is considered to be one of the 

^ following four types: (1) /HELP, (2) an error, (3) set-forming 
(SELECT, COMBINE, LIMIT) ,. or. (4) any other. Type 1 commands begin 

, with TCI, type 2 begin with tC2, type 3 begins with TC3 and type 
4 begins wdLth TC5. The command proceeds as follows: if the commard 
is type 3, it is. next checked by TC4, then TC5. Types. 2, 3 and 4 
are checked by TC6 and all types of commands are checked by TC7. 

4 

When any -threshold is exceeded, the appropriate in-depth analyses 
are called. When the in-depth analyses are complete, the comtnand re- 
sumes its path^ through the threshold checks. Thus, a given command 
could tliapretically exceed more than one threshold in its path through 
the preliminary analyses.' After the command has been subjected to 
all the appropriate threshold checks, cpntrol passes back to the shell 
of the instructional processor xAich called the PAR, 

3: In-depth analysis - general . In-depth analyses are designed 
to analyze data from thresholds crossed in the preliminary analysis. 
There are five in-depth analyses: (IDl) /HELP, and time use,. (ID2) 
error analysis, (ID3) zero set formation, (ID4) piuise and cluster 
analysis and (IDS) duplicate checking. Details for each of these 
analyses will be given later. 

When called, the ID enters the code^ for its c^ll on the command 
and control data file as well as recording the clock time of. the 
call. The clock time of the return is also recorded. Each ID h*i3 
a counter for the number of times it has been I'sed, time is logged 
to enable later analysis of the cost of PAR In ligkt of the benefits. 
When entered, the ID checks to see if it has been used more than T16 ' 
times before. If so, the proctor is notified and can participate in 
"^^^ student.^ In the case of passing the threshold 
tor /HELP use, the proctor is notified immediately and all negotiation 
ihn'h^"'? 7 "-^^ vrootov. . This seems the best way to help the student 
who has already extensively used the resources of the /HELP program 
con^?! crossings of the threshold does not exceed T16, the 

TaMI by one, and the data from th* preliminary analysis 

and the data files is analyzed to decide how to deal with the student's 
possible problem.. When the in-depth analysis arrives afa plausible 

^hf^ri -J^^ conclusion reached. The ID negotiates an agreement \^th 
acMof^K^ ''^ '^^"^^ °f threshold crossing Sd ^hS - 

Se ID ogs%1;e'retu'rif.J° ^^^^^-^on ot the L^^iS^on, 

1? ki ll' t ^° preliminary analysis program which called 

^lySs arr™r:"' the prejS:f^ 

old che?k In Se ^^!f^^ ^^.^^ continues to the next thresh- 

r^^n^n! I , preliminary analysis, or if all have been checked 

c^IS^i^? '° '° instructional processor ^ch ' 

ERIC , , 



/ 




Kigure 3#11« Order in which thresholHfj are checked by the PAR^ 




- 62 - / 

/ 

/ 

/ 

As a result of the in-^epth analysis, the program discusses /he 
student's search with the student. In the process of the discussion, 
the PAR can use any of the resources of IIDA. Another in-depth analy- 
sis may be called, some of the displays from the /HELP program can be 
used and text from the tutorials is also available. The PAR is in- 
tended to be flexible not only in detecting trends in the student 
search but also in presenting alternatives. The ultimate Judge of the 
search remains the student who may refuse the PAR suggestions without 
prejudice. Records of student use or rejection of PAR suggestions will 
allow for evaluation and growth of the system. 

Details of the dn-depth analysis . Each threshold can be 
crossed for several reasons. The in-depth analyses .take the data which 
exceeded the threshold value and adLd to it some data from the student 
performance data base. Certain calculations are made in the in-depth 
analyses to check for patterns in student performance. The path 
of the in-depth analyses depends on how each threshold was crossed. 
Discussion of each in-depth analysis will be divided into sectious'-by'^'^'' 
which part of the threshold check called it. ^^^^^'^'^ 

IDl. /HELP and time use 

A. If use of /HELP excee/j Tl% or T2%, notify the 
proctor 

B. If time between commands has been greater than 

T14 seconds T15 times, ask if the student regularly 
works at this pace. If not, offer /HELP menu. 

ID2. rrfor analys is 

A. Of the last T5 commands of which T4% are in erzor: 

1. have arrors been in one type of command. 

2. have errors been in the command words 

3. have errors been in the argument of commands 

4. none of the above 

B. Of the T3% errors in the use of one type of command 

1. have errors been in the command word 

2: have errors been in the argument of commands 

3. none of the above 

ID3. Zero set formation 

Of the last T17 zero set forming comm^inds: 
A - how many were formed using SELECT with: 

1. string search 

2. a descriptor not seer; in an EXPAND table 

3. a descriptor seen in an EXPAND table 

4. none of the above 
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B - how many were fbraed using COMBINE with: 

1. AND with small or zero sets 

2. AND with many sets 

3. NOT 

4. none of the above 

C - how many were formed using LIMIT: 

1. and, if limiting by accession number, failed 
to^use UPDATE to determine the correct number 
range 

2. and, if the set being LIMITed wfts formed 
using an identifier (ID), limited the set 
by major or minor descriptor 

3., none of the above 

ID4. Phase and cluster analysis 

A. If the number of commands la the current phase is 
greater than TIO; 

1. have all three phases been used at some time 
in the search 

2. if no, what phase or phases have not been used 

3. if yes, what proportion of the total nuiber of 
commands have baen in each phase 

B. If the number of phase changes in the last Til 
commands is greater than T12: 

1. have all three pha.ses boen used at some time 
in the search 

2. if no, which two phases are involved in the 
current phase shifting 

3. if yes, what proportion of the total number 
of commands hava been in each phase 

4. if yes, is there a most frequent order of 
occurence for the commands 

C. If the number of C^MBINE-like commands in this 
cluster exceeds T8: m cms 

^il tM^T^^ °^ ''""^ '° '^'^ '^^"^t sets 

2 in u"^''^'' converging on the student's zoal 

3. if yes, how many phases have been used 

for^elcft' tt ^^^'^^^^S^ °"^er of commands 
tor each of the other clusters 

l"ss1haT??r °' is 
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1** how many clusters have been formed 

2. how related are the clusters ^ich have been form^-i 

3. Is the number of items in the most rerent sets 
in the cluster converging on the student's 
goal 

4. how many phases have been used 
ID5% Extr aneous or redundant commands 

Are all the T13 duplicate commands: 

1. tLe same command type 

2. set-*forming commands 

3. nou'^set^'forming comoiands 

^ entered in pairs, so that the duplicate is 
the next command after the' original 

5. lacking in a pattern 

3.4 Tutorial Subroutines 

J 

In review, the shell c" each tutorial: controls the sequence 
J- of frames, changes the frame ntanbar parameter at the top of each 
frame, and controls, conditional branching from section to section 
within the frame, or to the top of other frames, using a code re- 
turned at the end of most sections. 

A section is a unit of coding wh^.ch accomplishes a single in- 
structional task as part of the larger instructional goals of the 
frames. .Three types of sections are possible, combined in variotis 
ways within each frame. They are: non-command response sections, 
coti.mand response sections, and program controlled command sections. 
Sections call a series of subroutines, usually Between 3 and 7 per 
section. Tlie characteristics of each type of section and of their 
subroutines are discussed below and are illustrated in Figure 3.12. 
Each of the three section types usu^aiy begins with a discussion ^ 
although this may be omilted or inserted at the end of the section. 

3-4.1 Non-command response sections 

This type of section most nearly approximates coding f^und in 
conventional computer-assisted instruction <eAI^. Text is i)rinted 
o response is elicited, the response is compared against several ' 
-.os^ixDle responses, the student performance data base is uj^kated, 
and a code is returned to the shell to indicace which section should 
follow. At the top of every section a text-group code is sett A 
discussion of the subject follows by calling fpr tlir. next-file print- 
out subroutine. The discussion usually ends with a r.^quest for stu- 
iSutf^^""^* ^^"^^^ ^''^''^ subroutine is ;,alled to accept the 

^exnH^^^c^^.-°^^^ ""^t expected responses may \e stored as either 
S^ii^i 11^^^ ^^^^ "J^^^ from the text file. 

In the latter case, the text-group code. is changed and a subroutine 
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which sets up arrays from the text file is called. The frame number 
aad text-group code together indicate a set of possible answers 
stored on the text file. If there is a match of the student response 
against the array, the index (pointer) of the response may become 
the code returned Co the shell for choosing the next gection* If 
there is no match, the subroutine sets the code to ♦rhe maximum index 
plus 1, ' In any case, the student's response is recorded on his data 
base for later analysis. 

3.4.2 Command response sections 

*^This type of section allows students to input '^ommauds which 
are Qent to the data base processor for a response. Text is printed, 
a command is elicited, the command is parsed and verified, it is 
passed on to the data base processor, th^ data base processor's response 
Is printed, the student performance data base is updated, and a code 
is returned to the shell indicating which section or frame should 
follow. Again, a text-group code is set at the" top of the section. 
A discussion of what is to follow may be printed here by calling 
the ta:ct-file print-out subroutine. This discussion will end with 
a request for student input, made possible by^ calling the studeot in- 
put subroutine. 

Command response sections will then pass to the parser (Sec- 
tion 3. 3 4.1) /the limits of acceptable input. These limits may specify 

'Che explicit command or a range of acceptable commands. 

/ \ 

When the command parser subroutine is satisfied that the command 
is acceptable, the command will be written onto the programs communi- 
cations-logging file. (See Sections 3.6.1 and 3.6.2.) The command 
will then be transmitted to the data base processor. The response 
will return to the program over the input communications-logging file. 
The response will be printed and parsed. Data derived from the 
parsing will be stored on the student *s data base. A code evaluating 
the input may be derived by the command parser and used by the shell 
to determine which section is run next. Alternately, the branch 
following this type of section may be unconditional. 

0 

3.4.3 Program controlled command sections 

This type of section allows the rogram to send commands directly 
to the data base processor. ' Responses thus generated can be very 
useful for up-to-date examples. Also, data derived from the responses 
can be embedded in the text to reflect updates of the data base ^hy 
the vendor. Text is printed, a command is issued, the data base 
response is parsed, and the response may be printed. Sections like 
this usually do not affect sequencing of sections within a frame by 
the shell. Rather, they operate usually in conjunction with one of 
the other types of sections in which branching is controlled. 

Again, a text-group code is set upon entering the section. A 
discussion of what is to follow may be printed here by calling thex 
text-file print-out subroutine. A command, either in nhe program in 
its literal form or taken from the text-file, is issued to the data 
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base processor through the communicafiiona-logging file.. Tha response 
is parsed and is printed or derived data is embedded in text. The 
text-group' code may be reset for a post-response discussion. 

3.4.4 Subroutines 

1. Text-file print-out . The purpose of tha text-'file ia to store 
data for tha tutorial programs. This data includes the instructional 
text, expected responses, and pr >graa parameters. (See Section 3,7,4.) 
The text-file print-out subroutine uses two params:ters set in the main 
program: the frame numbar and the text-group coda. The frame number 
is, used on the first block of the file to identify the frame ''s group 
directory on the file. The text-group code is used with, the group 
directory to identify the location of the appropriate text. The 

text is extracted from the file and. printed at the student terminal. 
After printing, the frame ntrnJaer, the text-group code, and the time 
since midnight are recorded on the communication-logging file. Any 
other applicable taj;ging data is also written. If the entry is to 
be communicated, a code received from tha proctor will have indicated 
this. The tagging data ia this case will indicate who is. to receive 
the entry. For a discussion of the communications-logging files, see 
Sections 3.6.1 and 3.6.2. 

2. Student input . This subroutine will accept input from the 
student, check for special system commands, and log all student^ input 
entries. Input from the student terminal is handled simply with the 
BASIC command INPUT. A small group of system commands are available 
to users of • the tutorial programs. These enable messag^ to be sent 
to the proctor^d quick log outs. Comparisons with permitted 
system commands will be laadte after each entry. . If a system command 
is recognized, the appropriate action will be taken by coding in 
this subroutine, including calls to yet other subroutines. Student 
input entries will be logged as either communicating entries or simply 
logging entries, depending on authorization from the proctor. En- 
tries are communicated to the concentrator and data base only after 
having passed through the command parser. 

. .1:. response set up. This subroutine is similar to the 

fltll P^*^-°"'^/° that it accesses text from the text-file using 
the frame number and text-group code. Instead of printing at the 
student terminal however, the text-g,:oup is read Lto an^rray of 

rtlttl^t\ T ^"^^'T^- ^« author is 

l°l ^T^^ responses in some order, perhaps 

be^ncoLrf Jh/^^ ^"^^ preferred. Generalized respo;se8 may also 
be encoded in this text-group, if the course author finds that desir^ 
able. The student input will be compared with entries in this a"a^ 
If a match is found, the index of the array .element couS serJe S 
?h!n L °^u'^\ °f response. This code^y 

If no Itch 1.%'''^"^''' calling, in- the next' section or ^Le. 
If no match is found, a code indicating this will be used instead. 

4. DBS interface. This subroutine will call both the r^r,^ 
and response parsers (see Section 3.3.1) as subroutSs and St 
manage posting data to the student data base anJ tJrSgSnglue 
After the command parser has elicited and recognised an Lcfptabi; 
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command , this subroutine will send the command to the data base 
processor through the communications file. It vill then print the 
response and call in the response parser. Data isolated by both 
parsers will be transferred to the student data base. The data 
base processor will, return a code through this subroutine to the 
shell, indicating how many times input was requested before an 
acceptable command was entered. 

5. Severe problem filter * Severe problems qf several sorts can 
be identified and referred automatically to the proctor. These in- 
clude; repetitive unrecognizable responses, exact repetition of 
responses, user-instigated infinite loops, and other "wise-guy" 
behavior. This subroutine follows the user input in most cases. 
Codes indicating severe problems can be generated in the commanci 
parser and picked up by the severe problem filter for transmission 
to the proctor. This subroutine «rill be able to send a message over 
the communications file to the proctor. 



3.5 Student Performance Data Base 

A file is maintained for each student which records each stu- 
dent's history of performance since beginning to use IIDA, with 
emphasis placed on the current or last search carried out. The file 
is attached to any program which the student runs, although data 
is largely gathered from activity with the exercise and the assis- 
tance modes in which the student performs real searches. The record 
I/O technique available under RSTS/E is used, with blocks of 512 bytes 
each. 

The first seven blocks contain seven directories for the history 
of the. student performance from seven different perspectives. The 
first block is a directory to a series of long-term data, • covering 
the entire association of the student with the system. .The next 
six blocks are directories to series of data on the current search. 
•These may be copied as frequently as desirable to off-line storage 
media, which can be mounted at a later date for research on student 
searching technique and behavior. 

Figure 3.13 shows how directory blocks are related to entry blocks 
containing data series. Each directory block begins with several 
bu 'is^^^lf general data about the entire sLies Jhis i Sudes, 
tory and the block and address available for the next entry The 

nd":L%^s\^:"hff in I' 'T'^'^ ofl^TlYoc^ n^.er 

and address in .hat block of each entry. Entries in the data series 
are found beginning with the eighth block. See the figure beloS 
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3.5.1 Personal and long-term data 

, General series data found on the first block includes both 
permanent data and data updated with each new search. Permanent 
data includes: password to enable use of file; personal identifica- 
tion code; affiliation code; total number of log-ins; and, for' each 
program available in th« system, number of uses in which the program 
was begun and number of uses in which the program was completed. 
The personal identification code is used in lieu of a name against 
a llat and is made available to only authorized personnel. The 
affiliation code, indicating class, company, or division, is similarly 
protected. ^ 

Data updated with each new search includes:' total number of 
searches; and block number and address of next entry. The' directory 

and1ddr!:r r "1" '"^'^^'^ includes the blocfn^be^ 

and address of each entry. The long-term data for each search re- 
mains stored and updated in the program's core area. The ?Ue ±s 

beganf tLe°\rs1cSi"takt'byihe s^'ch^o^i^^^^^ T^'^ 
t,,, .... c , uy tne searcnj how many commands hqpH 

by type of command; the total number of s«ta fn-r™^^ ^^""uu^nas used, 

(InideptW - -J^tl, ?H °* P^fonnance analysis callj 

data. complete dlscaaslon of the uso of this 
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3.5.2 Command and control data 



\ 



General series data found on the second block includes; total 
number of * commands indexed by the directory; and the block number- and 
address of the-next-possible entry. The directory to commands used 
in the current search, also found on the second block, includes the 
block nym^r and the addr^ess in tl^^t block of each command entry-. 

I^ja'-'AS^eaph entry includes; a code for entry t/pe (which 
search commanH7 which control command, an error, or a PAR in-depth 
analysis call); clock time when .entry was made, in seconds since 
midnight. Thfe following three data items may use the same relative 
address in each entry; for valid search commands, phase of command; 
for invalid commands, the error code; and for /HELP, the clock 
time since midnight at which the student returned to the main pror 
gram. In this last case, the first time entry is the time at which 
/HELP was called. ' . y 

The entry will also include the set numbef , if the conmaiid was 
SELECT, COMBINE or LIMIT The length of the remaining dafT in the 
entry is then indicated. :f the entry is a command^r^n error 

1^'^'^ ''^^ ^^"^"^ °^ entrrls /HELP, this 

will be the sequence of selections made from.^l{e menus in /HELP. 
If the en*Ty is an in-depth performance arfalys is, this remaining 
data will oe an encoding of the activity within th6'PAR._- ^ ^--^ 

■ When histories of valid commands are displayed, all- other dom- \ 
mands in this series ^errors, control commands, /HELP calla, (and 
fAR calls) will be bypassed. For histories of errors, all odhet 
commands will again be bypassed. Integration of commands in this ' 
series allows for reviews in which the sequence and position of 
various types of entries is of interest position or 

The command and control data series is updated with every search 
paSr 'ifthf "T'-. -"""^"'^ '^-^ the co2an" 

L^s gnLM^e ^^^^i:^:^^^^ F'^S. 
is retumprf h^r^yZ I search phase analysis. The set number 

3.5.3 Sets created data 

General series data found on the fh-frH j -i ^ 

number of sets indexed by the dSec-tor^ and tSe b^"\ ^ 
address of the next possible entrv Si ^^^e. block number and 
the set numbers themselves with en^riS inlh^ M '"^^ 
one-to-one with the set numbers inlL^JLct ^^^h^^^d^^t-rTto's-et':^ 



creatad in the current search, also found on the third block, in- 
cludes the block number and the address in that block of each 
set entry. 

Data on each entry includes: the number of records in the set"; 
the code for the type of command that created the set; the cluster 
to vr«ich the set belongs; the length of the original argument; the 
text of^the argiflnent. If the set was created with the COMBINE cora- 
ntf,nd, the lergth of the argument reduced to elementary set munber 
is given, followed by that argument; and the length of the argument 
reduced to alphanumeric descriptors is given, followed, by that argument. 

COMRtS?^ ^^^^r^^"^^^^^ ^^^^ "''^^^ updated with every valid SELECT, 
COMBINE, or i.IMIT command. Tlie number uf racords in the set coues 
from the response parser. The command type code comes from! the 
command parser. The cluster number comes from the last run of the 
cluster analysis routine. The text of the argument comes fr'om the 

suZnf/"!"' i« COMBINE, the argument reduction 

subroutine is run and the two reduced arguments are stored. 

3.5.4 Sets viewed data 

General series data found on the fourth block includes- total 
number of sets indexed by the directorv- «nH ^ -^ncj-uaes. total 

- ^, •' '■'"^ "J-reccory, and the bli;ck numuer and th*» 

address oi the next possible entry. The directory entrlea correso™^ 
one-to-one „it!. the set numbers, the Ic .etlon ofThe d^rectorrd^e 
for ejjy. given set can b. calculated from that set n^er Se 
■ tartS b1^o,L '^he current search, also f„;;d S the 

set'etf-y ™^ ^''^"^ °" 

an re°":ds°tn%t':i?'tiat'Lie\f°"' f "^"'^•■S 
eludes: the rer,rd nJioJer I'X se"-' hllel °" 
recordj.th, for^t In , hich Jhe e ord ^as seen-'^^d' the b? 'f'V'" 
address of the n^vt T•o^.^^J j ^ -r seen, and the block and 

IS i^^^^z Td;. act;5 ^r?^ ^^VZ' 

.cKain.Vnserting ihe secLd vlft^!*! ? by tracing through the 
•on'the un. *=*edUte3^^e'c:SrthrL:r1l„'r"'"^ 

cpi^nt 'w'^'rL'mb^1\nTf:^^^^ ""'T' ^^^^ 

parser.' The recorfnu^bet L iJ^ia^Sr.'^^^ ""^^^^^ 

is J&remei^ted by 'I'" for every sub So the coimuand parser and 

th'^student fdr the relevance of eacJ recoS! "^''^^ 
h^"^^ Records viewed G^lta 



of the next possible entry,. The direccory to records viewed, also 
found on the fifth block, includes the rscord's accessidn number, 
the block niunber, and the address in that block of each record entry. 
This directory is dynamic, that is, after each new entry it is reor- 
dered by inverted accession number sequence. This simplifies searching 
for the entry,- which is necessary for each record viewed. 

Data on each entry is in tlxe form of a linked list referring 
to every time any given record has been viewed. Data on : this list 
includes: the set number from which the record came; the record 
number in that set; the format in which the record was viewed; the 
relevance assigned to the record; and the block and address of the 
next entry of data on this record. 



The records viewed data series is updated with every valid 
TYPE command. The accession number is isolated by the response 
parser. The set nurher and format number come from the cdmmand 
parser. The record number is initialized by the command ^par jer and 
is incremented by *1* for every subsequent record seen under a given 
TYPE command. Relevance comes from the routine which asks the stu- 
dent for the relevance of each record. If a match of accession 
numbers occurs in' the search, the chain of entries must be traced 
through to the end, at which point the new entry may be posted. 

3.5.6 Descriptors used data 

General series data on the sixth block includes: total number 
of descriptors indexed by the directory; and the block number and 
address of the next possible entry. The directory to descriptors 
found in the current search, also found on the sixth block, in- 
cludes the block number and the address in that block of each descrip-- 
^or enSes ^^"^^^^ alphabetized according to the descrip- 

.^Wnaf'^K°^^^''^^''''^ includes: The length of the descriptor \ 
string, the descriptor string itself; the number of times the des\rin- 

and l^llv S''T''f ^^^^ "^"^^ (thesaurus access) 

and COMBINE (derived from the alphabetically reduced COMBINE) ; num- 

The descriptor aded data series is updated with pvprv „oi-t^ 
SELECT. EXPAUD, or COMBn.E =„™a„d. The d«cr pto Ifjsolatad 
by the coMnand parser. IC the command Is COMBlk, the argument red"- 
tior. subroutine Isolates each argument In the alpianujneiic fc™ 
llltf' l"""^ i"^""'" "l^'^^'i "-^^ from tSe rSionse 

rs^htfsiughTif s T.^'^vrT: 

be posted to tL ..^^ ^^^ti il^-d-rn^in^rT^^ 
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will. be placed at the end of the entry block, and the dir'jctory entry 
to the entry will be inverted in its alphabetic position, 

3.5.7 Expanded index data 

Expanded index data is essentially the first and last term found 
in every EXPAND-table. This data helps in det-»naining if a descrip- 
tor would be seen on an EXPAND-table," if it is the index. General 
series data found on the seventh block includes; total number of 
descriptor pairs pointed out by the directory; and the block number 
and address of the next possible entry. The directory to the pairs 
so derived in the current search, also found on the seventh block, 
includes the block number and the address in chat block of each pair 
of descriptor, entries. 

Data on each e. :ry includes: the length of the string repre- 
senting the first descriptor; the first descriptor string itself- 
^v^Ax^''^''^"^ ""^^ ^"^""S representing the last descriptor in tb/ 
EXPAND-table; and the last descriptor string itself. The directory 

^l^h^LM°T f "^S: ^"''^ ^° '^^^ f descriptors are in 

alphabetical order. The expanded index data series is updated with 
every valid EXPAND command. The two descriptors stored are isola ed 
oy the response parser. ' - 

^•■5-8 Transition from oiock to block \ 



b1o.J?^^^^' i?'" °" ""^^ ^"'^^ ^1°^^^ is raservad a next- 

seri s ?s Stored if "^^^ -"-t 

sen. 3 is stored in these two bytes. Before an entry is actually 

I " "'"^ data 13 placed in the block 



3.6 The Communications Process 



or 



cesser, the vroclorTTt^^lT^Tl^tll'sl'^' 'S!""!"™'^ l"' 
sant from any of these to any other one ?he ""'«8" ""^y 

grams which physically control thr,,™!™ The set of computer pro- 



g- «^W«Ub < 

ommunications processor. 

The others are: 



-sage" o1g\S ^--i-' performed. The others 

tween'the cf an^th^DL Ld Ml^r '^""""'"'^'"S ""^^^^8-- ^e- 
provides infonnatLn of ^se L anaivzinf ^"^^^^ 
needed to spot and correct .o^ttl^lS^-^^ b\T ^stTt; 
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expand * without creating bottlenecks. The proctor hag some functions 
which require IP-like programs, but in most cases, he is werely using 
communications facilities to send messages to students, receive them, 
or monitor a student's activities. 

The concentrator, although only a part of the CP, plays an impor- 
tant role. We have noted previously that the actual percentage of time 
a line between a student and the DBP is in use is relatively small, 
expecially for inexperienced users. The concentrator provides a 
means to reduce the cost of this usage pattern by making fuller use 
of the CP-DBP line, shared among multiple stud-^nt users. 

The billing program monitors the amount of time a student is 
using the system and computes its cost. 

3.6.1 Message switching 

. Communication between simultaneously operating programs is made 
possible by message switchers. These programs manage pairs of input- 
output files from all programs operating at "any given time ±i\ the 
system. Any program operating under RSTS/E and written in BASIC- 
Plas may have at most twelve internal channels assigned to it. vSee 
Section 3.1.4.) Files on these internal channels may be shared be- 
tween programs, although only one program is authroizfed to write onto 
a file so as to extend it. / 

Each message switcher may have files assigned to as many as all 
of its twelve channels 'as follows: one to the prot^tor; one from the 
proctor; one to the concentrator; one from the concentrator; one to 
each student, > to four students; one from each student, up to four 
students. As many as four message switchers, accomodating sixteen 
students as a maximum^ vill be considered at this point in the desien ' 

Xl"o3^S:? '"'.^^ "'""^ '"^ concentrator !r.; ' 

These communications files are handled by the record I/O tech- 
nique available under the RSTS/E operating system. These files also 

L'nrJ^f '""''^^ ^^^^i- 3.6.2. Each file be" ' 
gins with a header record. This record Indicates the fol lowing- the 

ill exerMse' ''"^^ "^'^ ^^^^"^^ progral ufeS'ctu 

nutVHr assistance), and whether the file was input or out- 

The following, referred to as taegine data „^-r.r-cA ,.-^u u 
messasp nn 4 ^ , '•"ftft.LUK gata. is stored vxth each 

message on both the input and output files: 

. message type code (see Figure 3.15) 

. total length of remaining data, including message 

. number of destinations 




• destination codes 
. origin code 

• the time since midniglit, in seconds 

• the message itself 



Certain iressagcs may be encoded, specifically, messages to the 
program and logging entries from the text-file. A time tag ;ts as- 
signe4 to a variable when the tagging data and message are stored on 
the bl^ck in the buffer. This usually takes place after the appear- 
ance orf entry of the message itself • 



Coming to the instructional program ; 

. message to be printed at the stWent's terminal 

. message to be used internally by the program 

J Going from the instructional program ; 

. logging entry - an .-^ncoding of a^textr-flle. message 

. communicating entry - an encoding of a text-file message 

. logging entry - verbatim message from the student 

. communicating- entry - verbatim message from t.he student 

• logging entry « - rbatim message from the program 

. coumiunicating entry - verbatim message jrom the program 

Figure 3.15 Types of messages onTh^ comiunications-l^^eeinc 
files, trom the perspective of the instructional progr^s ' 



d^.rntc/^'^''"''''''"^''''^^ program" will be defined for the following 
discussion as any tutorial exerr-fop ^t- i-^j-j-uwing 

acts directly wi?h a stulnt a^f tem^L^ ^rorr.'"'""" ''^^'^^ 
structional programs, the firs^ taeeW d^;../ Perspective of in- 

£ , J-irsc tagging datum, messaffe-tVDo code ■^r./^^ 

cates one of the conditions described in Fieure 3 1 c^%^^^^ ""^^j 

in I tutori;! ^rogrS? ' '^"^ text-file 

bytes. These bits " Sponf olL^ x L°u '"/^^ 
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that the file on the channel contains information for the associated in- 
structional program; the other bit indicates that the file on the chan- 
nel contains information from the associated instructional program. 

Communication may be viewed from either che point pf view oc a 
typical ins true tiorxal or proctor program or from the. point of view of 
the message switcher. The output file of each instructional and proc- 
tor program is an input file to ^ne of the message switchers while the 
output files of each, of the message switchers are unique input files to 
instructional and proctor programs. The logical framework of the mes- 
sage switchers is shown in Figure 3.16. 

1 . Communication from the point of view of the instructional programs , 
l^hen a program has a msssage to send^ it proceeds as follows: It reads 
into the I/O buffer the block which contains the last message sent. The 
block number had been previously saved in a core variable. If the mes- 
sage is too large for the space remaining in the block, the next-block- 
indicator is turned on, the block is vnritten out, and the next empty 
block is called in. • 

In any case, the following tagging data is written onto the block: 
the type of communicating entry; the total length of the remaining 
data and the message; the number of destinations; indications of the des- 
, tinations; time since midnight; and the message itself. Tlie T^lock is 
written to the file and the appropriate bit in the common memory area 
is turned on to indicate that a message is to 1)e transferred. The 
^!i^'f''>u^'^'^''"' location and, if necessary, tue block num- 

fn^ ^ output file are both updated in the core variables reserved 
tor this information. 

, ^^Jt^ program may either do some processing or may "wait around" for 

checks thrinS:;n. ° P^S^^"" -^^^ - 1°°P which 

'r^^n .ww?^ T^'T ^^"^ Otherwise, the pro- 

oram will check the incoming message bit when the oth(>r processing 
completed 'Aen the bit is turned on. a message is waiting for the pro 
gram on the program's input file. waic.ng ror the pro- 

The block of the next input message and the location of the next 
message on that block are also stored as variables In rhl ^Ll rn, ' 

the input bit is recognized by the program, hi prog?a^' u?n:'ir;ff^" 
The appropriate block from the input file is then retSeved if 

&cige IS excracted and used appropriatelv THp var-fo^^i«« 4 
blocK and „uuv. .uJsl Z the bI;cK'^^e";\':"racco"";:j?. '"-^ 
2. Communlcaulo n from the ooint nf ir-fow •■u 

gra.. wants "".aid^L.lge" A J« m '""""i"""! Proctor pro- 
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next program wants to send a message, etc. After chRcking the last 
Dyte and switching any messages from the associated file, it will re- 
. turn to the first byte in the "common memory area (See Section 3.1, A). 
The message switcher will iterate this loop' the whole time the system 
is operating. •' 

^r.A, ^ indicates that a message is to be switched, variables 

^^fi" '5^^^°'^^ relative address of the next expected mes- " 

M^ ? ^^^^""^^^^^ "i'^h the bit are used to access that message. 

S^e blocJ'ol'^H "f.^"""'"^ "^^'^^"^ '° transferred is turned off Sd 
Sv oiret 1 ? . '^°"^sponding to the bit is read into the buffer, 
tde ?f L.^^i"*^ ^^'^ recognized by the message-type 

inpd:t:d':s^;eed'^d."^'^'^''^ '"''^^^^^^ ^^^^^s 

tagglS'dftr''?ab'?^^ °' r'"^" are detemined from the messages' ' 
for each coordlLt^.. ""'"'^'^^ ^^^"^^ stored 

data enatlL canine inlh^ message-switching program. This 
sag. switch" output n Alaln^'ifthe'"'^' °' the appropriate mes- 

' ^J^'X^^'^l - - -^l^t-to^^fi^dlSo^t- 
turned on, the block xs written out, and the next empty block is called 

bio,k'Li;JtLr::t To\^:Tii '^'^'^ ''-^i ^^-^ 

memory area is turned on Jo indi;atrtharr' 'f' """"^ 

associate instructional orLr^! ?f f message is waiting for 'the 

of the next message on this if' \ I . "^^^^'^^ ^d^"-- 

variables. ^ updated . -'n the message switcher 

relati;eir?igo7f^- ^o^diltinT I\'''t ''''' '^"^ "'^^ P"^^- 
name. This nSe ^u^^ bl associated with"' '^"^ ^ ^^^"^ ' 

and a common memory bit as Lng as the fiLrfxilt "'J''^'"' ^ 
necessary to identify this association of f^lpe """^^"^^^Ping is 

and to keep track of what channels are ?ree on whL '"'''^'"^ " 

tree on what message switchers. 

A single block of a commonly accesslblp f-n«, ^ 
memoi-y, and a few lines of code <n ^""'^ of common 

.fice for housekeeping. The coWn fn! uV^f^^ switcher program suf- 
tured to associate the nl^es TSl currentln''' ^A^^^^^'^'^y ^truc- 
sage switchers, and the bit and channel d^lf ."' ^^'^ "^"^ °^ ^^^e mes- 
the message switchers. devoted to the files on each of 

When a program is lopged on t-n 
by the login or reception'program If aT''"' '"'^^ ""^"^ 'Checked 
pair of channels, the program's file ''^'^ switcher has an open 

structure and a bit in comrnJn i °" written into the 

switcher. Along with t^r^^eC fTL b. f'^"'' °" " 
jally by the message switcher - checked periodi- 

will call in the common file block r^adS^!^ ^"^"^ -^^e switcher 
Of Ules, and attach the.ew ^^^^s\T:L%^^ 
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switcher then tuxma off the bit. Whenever a" job ia detached or logged 
off, the logoff program eliminates tl'e file name from the coimuon file 
block and turns on the message switcher's bit. When checked by the 
message switcher, the common file block will indicate that those chan-* 
nels are no longer required. The message switcher closes those files 
on those channels and turns off the bit. 
< 

Proctor Intermediate message merger . Problems arise In se- 
quencing messages to and from the proctor (see Section 3.6.3), since 
up to four message switchers are sending data to and receiving data from 
the proctor program over as many as eight files. If these were directly 
attached to the proctor, only four channels would then remain for all 
other files used in the proctor programs. The concentrator (see Sec- 
tion 3.6.4) is similarly limited. It, however, tan tolerate eight 
channels tied up in communications. The proctor cannot tolerate this. 
Therefore, an intermediate message switcher is provided to merge the 
four input files into one input file for the proctor and split a 
single output file into four output files for the message switcher.".. 
This reduces to two the number of channels allocated to communications 
for the proctor. . 

The intermediate message merger operates as follows: The message- 
switcher-to-proctor bit is checked for each message switcher in se- 
quence. If it is found on, messages are read off the file and written 
to the single output file to the proctor. (Input file from the proctor's 
perspective.) Added to the tagging data is a code indicating which mes- 
sage switcher sent the message. 

Alternating with this process, the intermediate message merger 
checks a bit in commoi. memory turned on by the proctor program when a 
message is written to the proctor's single output file. The message 
and its tagging ckta are read." The intermediate message merger deter- 
mines from the tagging data which message switcher is to recline the 
message. The message is then written to the appropriate file 

JnScat?^°M 'i'' T ^''f °" ""'^li"^^ 'variables 

indicating blocks and relative addresses are maintained by the inter- 
mediate message merger, also as outlined above. 

3.6.2 Message logging 

Messages are logged on the same files as are used for communl 
eating messages. (See Section 3 6 1) To rer^n^^ T . communi-- 
has associated .1th it an input file nd L o^t ut fi e'' 'Sself^ 

Zll~'s)l Zr^T' '1 "^^"^^"^ avalSL'with Ba'5i?!pi'^^^ - 
unaer the RSTS/E Operating system. The program's input file Is sf-T•^n^^„ 

teminal. ihe co«,.„lc,ti„„. taction, arc ^sL.^ed Jn S^ctl™ "'e?!: ' 
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the logging functions will be discussed here. 

Messages are logged on the output file following every message 
entered at the terminal by the student and followi. g the nrinting at 
the terminal of every message which originated with the program o^- 
with the text-file. Messages from /the student and from the program 
are logged verbatim while messages whose source is the text file are 
encoded. 

All messages logged are prec'eded by tagging data. For verbatim 
messages, this includes: 

. code indicating that message is a verbatim logging entry, 
and indicating whether it originated with the program or student 

. total length of remaining tagging data, including message it-' 

. the- origin (student program) code 
. time since midnight in seconds 
. actual text of message itself 
. For encoded text-file messages, tagging data includes: 

. coda indicating that message is an encoded logging entry - 
. total length of remaining tagging data 
• origin (student program) code 
. time since midnight in seconds 
. code indicating whic'- ext-file is in^^olved^ 
» . frame number and text-group code 

data."?;: zii :j^L^r:es^sfS! i^'^^ 

eating entries as well as slmnlv iL r ' encoded as communi- 
the proctor can monitorltud^nt'aitfvitl 'dLl"' 

try is to be communicated desMn^M "^^^'^ the en- 

mal positions in the tagging data! " " '"'^'"'"^ ^^^^^ ^or- 

The logging-communication files ran he j 

flow, reposition students after a system '° "^^^^^^ 

in real-time. Each of these function^ J ^"'^ searches 

of programs, will be discussed below? ' ProgTa.s or parts 

--ons I^^S^tirS^^ in b.tch mode after 

the program which indicate thp i J ^^^'^'^hes are set at t'.e top of 

'e run. one .~T,toZ r^'^tlZl Z^lt ""^ '° 
/ . ^ ograms, or all programs which have 
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operated during the day. Response times and activity demands can thus 
be evaluated from the point of view of a single user, a group of users, 
or the entire system. 

The output of traffic analysis Is a report tabulating the following 
data for each message: time in seconds since midnight, time in seconds 
since the immediately prior entry, user or program number to vhlch this 
entry belonged, and type of activity this entry represents. Type ^>f 
activity may be displayed as a check in one of several columns or as 
a code number. Types of activity include: a text-group from the text- 
file, text originating in the program, input from a student, communi- 
cation from a student to the proctor, communication from the proctor 
to a student, message sent to the concentrator, or message received 
from the concentrator. This data is derived from thfe 'message's tag- 
ging data. 6 6 

Tb.e traffic analysis program will process the logging files as 
tollows: Up to five pairs of input-output files will be assigned ten . 
channels in the analysis program. These now are the input files to 
the analysis program; A^working output file will be assigned the pro- 
dlT. win hf . "'^'Sin, destination, and function 

Lhe input files into an array. Data from the oldest message will be 
reformatted and written to the working output file. Tagging data for 
the aext message in the same input file wiU repla;e thf data wMch Ls 

writterand'^; '^'T '''' ""^ ^^^^ rSo^a^^ed 

ana written and the process is repeated. 

<„ -i," 'f logging-communication files are being copsldered 

cess oaainel iSove'if^e^teS Cln 1o ll?'"^' P'- 
tion files, creating a seLndIo klng'o^tp" f ^tltir"""^- 

be reSe"t .T^^ti {\ 't^!^^^^^''^ 

now used for takina'd- . " analogous process is 

Data about Lssag!s^s p" L;;"",^^'^h: 'iJ^' ^^'"^'"^ 
differences between the consecutive "e^.- ?° 

printed as describe.^ aoo" calculated. Output is 

tlonin^ slSi^frf^fgl^^^^^ °^ the logging file for reposi- 

The lodging fi^es ar us«d VI T . " discussed in Section 3.8.3. 
aata bafe I. ^^s^^^t^JT^^^^^^^^^;^^^ 

monito; sfSi^'Sfllfffg^^ . P"'^^- is able to 

students prog^a^'IS ^^s^g ^^o^^a^'l'"^ ' "^^"^^ ^° 

the program to communicate a copy of "ver-- L^f" switch, which directs 

copy or ever/ message, except those generated 
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and sent by the jfrogram for internal purposes, to the proctor. The 
message switcher is responsible for sending to the proctor a copy of 
all messages going to the student* a program from other programs such 
as the concentrator. Tlie student is aotified by a message at his ter- 
minal that this monitoring is taking place. 

3.6.3 Proctor functions 

The proctor has access to a number of programs and capabilities. 
These include: reviewing in\depth the performance of one student at 
a time; communicating with any student at any terminal; monitoring in 
real-time all activity taking place at any student terminal; summarizing 
the performance of any group of students; accessing the loggings-communi- 
cations files of any group of Students; and triggering an in-depth 
performance analysis to be run on any given student. 

1. Reviewing performance . This option is based upon the /HELP 
option available to students. For a discu^siotr^f /HELP as a student 
option, see Section 3.2.4. The proctor has ^re options available 
here than do the students. The proctor may see the history of com- ♦ 
mands issued in three modes while students may see it in only one mode. 
These three modes are: valid commands; valid and invalid, commands in-- 
termixed; and valid, invalid, and control commands intermixed. 

The proctor can furthermore .race the entire history of any stu- 
dent's use of /HELP. This data is recorded on the student data base 
as detail on /HELP commands in the command and control data series. 
After seeing where each use of /HELP is located in the context of all 
the commands issued by the student, the proctor can trace the history 
of all options taken and responses made within- each use of /HELP. 
These reviews can be selected by referring to the /HELP command's 
sequence number in the history table of commands given. 

2. Communicating with students . In programs under proctor con- 
trol, the function /SEND is available. With it, the proctor can send 
a message to any student program operating in the system. This mes- . 
sage is transmitted via the proctor's output communication file. 
/SEND can take as its argument the names of all programs which are to 
receive the message and the types of messages they are to receive. See 
Section 3.6.1 for a discussion of message switching. 

f 

Two types of messages may be communicated to a program The 
first is to appear at the student's terminal for the student to read 
and respond to; the second, is a message to the program to set an in- 
ternal software switch. The two functions are distinguished and con- 
veyed by the tagging data that precedes the message on the communica- 
tions file. 

f.,n^^ "^r^".^ °^ ^"'^-"g ^ message to the student's terminal may be 
fZlJ" monitoring of student activity. If the proctor 

recognizes -that the student needs a particular type of help, he can 

, * 
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send the student advice with /SEND. An instance of sending a message 
to a program may ba found in the proctor's triggering of an in-depth 
performance analysis on a student's search. This authorization throws 
a software switch which routes control around the threshold checkers 
into the full analysis. 

Periodically in the proctor's programs, a subroutirfe will check 
the proctor's input buffer for data, ilf that data is found to be- 
gin with /SEND, the subroutine will parse the input for further informa- 
tion: to whom is the message intended and what message is to be sent. 
Messages to programs will be encoded. A list of this encoding can be 
posted to a text file for the proctor to review if necessary. The 
subroutine which checks the proctor's input buffer will look like this 
in BASIC-Plus: ' 

INPUT A$ 

■ WAIT n (a second or two) 

ON ERR m RETURN (no input) 

(process input found) " . 

RETURN 

J- Monitoring real-tiu ^. activity. This function has been dis- 
cussed already in Section 3.6.2. from the perspective of the logging- 
connnunications files. Student activity can i be monitored by the proc- 
tor as tnat activity happens in real-time. Proctor monitoring may 

■ iork arL%'li : '"^f^ " i"'^" the procLr's 

work area following a recapitulation of the student's current search 

f°Ls^ ''r'°J- h-8i-^ with the proctor's progrL sending 

a message to the student's instructional progtam. The message sets 
a software switch in the student's program which causes eve^/m^ssage 

to the proctor. A message is also sent to th^ student at his terminal 
notifying him that monitoring of his activity is beginning " 

;=!;::.£ 55.: 

t 
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dicate at the beginning of- the program what group of students the re- 
port is to include, and which type of summary report is to be generated. 
The performance data bases of the group indicated are attached, one at 
a time, to extract data from the long-term series and print the re- 
quested information. No information about particular students, 
identified by name or number, is releajed. Data on the dist.^bi'tion of 
each characteristic is accumulat-.d for a grand summary at the end of 
the report. Data presented in these reports-Includes calculations made 
from data extractea from the performance data bases. Data- presented in 
the reports are listed in Figure 3.17. , ' 



^- Agcessin)^ the logging - communications f^^pc^ ■ The proctor can 
run segments of the traffic analysis program on the^activity of any group 
of students. See Section 3.6.2 for a con,plete description of the traffic 
analysis program. The report will include the type of entry, its ori-in 
^d destination, the absolute time of the entry, and the elapsed .tiie^iTce 
the previous- entry. Suuuaries of the tim4 distribution of different types 
of entries are generated. ' 



HEADER DATA 

. identifier 

. search or summary 

. if detail, when was- search made 

. curreit searcK indicator 
COMMAiNDS ISSUED 

• total number of commands 

. total number of command types 

• tote number of commands by type 

. distribution of time between commands 
SETS CREATED i ^ 

. number of seta created 

. nu4er of sets created per 'command 

. distribution of time between the creation of sens 

- array of commands used to creat- sets 

tne proctor (.dontinuad on next page). 
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DESCRIPTORS USED 

. number of descriptors 'used 

. number of descriptors' used per command " f 

. number of descripliors used not found in EXPAND tables 
. number of descriptors used in free-text SELECTS. 
RECORDS VIEWED 

■ ^ntiffiber of distinct records viewed ^ 

■ 1 ' " - . 

. numb^ of records viewed, by format ■ 

. distributi^jn of relevance ratings; 

. number of records viewed out of total records in allots 
ERRORS MADE 

. number of errors made 

. number of errors made per command 

. number of errors made, by type of error ' .. 

/HELP USAGE _ - 

. number of /HELP calls 
. number of callk per command 
. number of selections of each option 
. absolute time in /HELP options 
. to||l time in /HELP out of total time search 
. total time in each option of' /HELP • 
P^RMANCE ANALYSTS Ppnr.PA>. yr^^^^ ' 
. number of PAR calls 
. number of calls per command 
. number of calls for each type of threshold " 
. total time for in-depth PAR - 
. total time in PAR out of total time in search 
. total time in-depth by each type of triggering threshold ' 
(Figure 3.17 continued) 
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^' Triggering performance analysis . The proctor may 'at any time 
send a message to any current instructional program in order to acti- 
vate the in-depth performance analysis routine for that program. The 
message will throw a software switch which bypasses the threshold 
checker, unconditionally activating the in-rdepth perfo'rmance analysis 
program. The message is sent over the communications file.' The ' 
proctor may want to do this whenever he observes a student with a 
problem which is not otherwise caught by the threshold checkers. 
« See Section 3.3.4 for a discussion of the" performance analysis routine. 

3.6.4*^^ Concentrator 

« > 

T.:e purpose of the concentrator is to- allow several students to 
use a single line. to the data base processor (DBF). Thus, the IIDA 
system, will look like a single terminal to the DBF 'and e^ch student 
will have the. impression that he or she has ,a single direct line to 
the DBF, The general flow of the concentrator program is presented 
in schematic form in Figure 3.18. . < . 

From the point of view of the concentrator, there are three types 
?o^^r^S2^' "'^^''^ ""^^^ ^° ^^"^ '^^^^ conversion on the way out 

h/J I °" ""^^ ""^^ ^^""^ ""^^ DBF (e.g., EXPAND); -those 

liJ^'^ho^! K?\""'°^f conversion on the way back only (e.g.\ SELECT);, 
and those which need set. number conversion both on the way out and on "" 
the way l?,ack (e,g.,. COMBINE). Four data tables are maintlin^d- by' the 

IT-lTTolll ^^^^ ;conversions. These arfd^scussed 

in the following paragraphs as Tables A through D. Table D will be 
further amplified in terms of its constituent rows. - 

Table A . This is a list of user Identif icat^o» numbers, posted 
for each command in the order in which they are- sent to the DB? 

S to^\r s ° e p'on^^Jo'LL'eT '^T'"^^ ^ 
to the correct outpu^ f iLs a"o' is pLj.d'o Tibirf f^'^n' "^^"'^^^^ 
originated with the concentrator A o . ^' command 

tion number indicates ^J^f^fJ .; / °" ""^^ identifica- 

sion to user set n^^bersli e S"' " f ' "^"^^^ 
LIMIT). numbers (i.e.^ the command was SELECT, COMBINE, or 

DBP set number is held as a pogtS to this list 1% 

spond^f—; seTn'ulbers"n?Lf' — " 

fication numbers' ?^ble C S rUt "^^ correspond to user identi- 

DBF set numbers bWore oL^ds ke cSraT^T^^i '""'"^ '° 
the DBF. Pointers ^to the last ^ar COMBINE and LIMIT are sent to 
of .Table D. . ' ^^^"^ ^^"^ """"^"^ a" f°vind in the third row 
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Table D . This is a matrix whose columns correspond to. user identi- 
ficati'on numbers and whose rows have the. following different meanings: 

Row 1, Output channel number . This may be an integer between 
•l* and '4', corresponding to the four message switchers. This data 
is picked up from the first message from the user on the corresponding 
input channels- 

Row 1. File number . This is the DBP file number presently 
being searched by the user. This is the default file number until 
another file number is picked up from the argument of a BEGIN or 
•FILE command. 

Row 3> Last user set-number >> This corresponds to the set 
number of the very last set created by the user. It is defined for each 
. column in Table C ^by the last row with pertinent data. It is incremented ^ 
by 1 for each user when the user creates a set. ^ ^ 

% r ^ ^^^^ 

Row 4. Count of r esponse destinations . This Is tak,en from the 
message tagging data arid Is checked with message tagging data that 
accompanies eadh new command. It Is used In formatting the tagging 
data returned with each response. This number defines for thri cur- 
rent column In Table D the last row with pertinent data In Table D. 

Row 5 and following. D estlnatl>)ns . The remaining rows in Table 
D are codes for response destinations. As in the fourth row, this is 
taken from tagging data ^accompanying each command, is checked with 
tagging data for tfach subsequent command, and is sent with tagging data 
• returned with each response. ' 

Concentrator processing will be discussed below, closely following 
the flowchart of Figure 3.18. Each numbered paragraph below refers ^ 
the process box in the flow chart with the. same number. 

^^lJ'JT^l''^^~^°~^'^'^ handler. Each of the four message 

Sse f iles'^th °' -'^"-ication files for the concentrator. 

These files with messages, or commands , destined for the data base 

LL'S\°uSod"d\s"""T^'^^^^' using the ^olon 

memory bit method discussed in Section 3.6.1. When an entry is recog- 
nized, its tagging data and the message Itself are rSd bj^he conc^n- 

as tne user Identification number. This will be used to 
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check Table D for the user's current DBF file number. If this^ differs 
from the current file number, a .FILE command with the user's file 
^ number will be sent to t;he DBF. This and all other concentrator-ini- 
tiated commands will be noted by an entry of '0' in Table A. 

Th6 tables are updated as follows before the message is sent to 
the DBF: In every case, the user number is added to the end of Table 
A and the pointer to the last command is incremented by-'l'. If a 
set is expected in response to the command, the user's last set number 
xn Table D is incremented by '1'; the last DBF set number in Table 
B is incremented by '1'; and the last user set number in Table D is 
stored in Table B. If the cbmmand is of the third type, with set num- 
bers that need changing before going to the DBF^ the commarid is parsed 
and, the set numbers from Table C are substituted for the set numbers 
in the command as given, making the command compatible with the DBF 
point-of-view on sets For the last two types =of commands, the sign of 
the entry in Table A is made negative to indicate that the set number 
m the response must be converted. 

3. Set up commands for DBF. Each command will be written to a 
Sorv^orthf file contains no tagging data, although a direc- 

the fne T^f w tne location and length of each command " on 

the file. The DBF message' flow control job learns from this bit v 

JSL^'T^ "^^''^"^ ^"'^ °" ""^^ ^"51i^ fil^- Action typically 

taken with respect to this bit is discuss^n detail in Section s's 1. 

bit ihVo^on"''' ^^"P°""P f^"^ The concentrator checks a designated 

fkL^n^Z ? 1° responses are waiting on the queue 

WtrrJ.^ AA f """^^Se flow controller posts Se 

length and address of all such responses to the directory for the queue 
tile If responses are waiting, the directory is checked " Waitine 

itiTToLrf/iti '''' t^r;rd!f i?^hich 

A is L ck\il:rt"is"" '?d r^.;J:v'V'i '^''^ 

^^^^^^ i™L 

set number is sub.Sitel ^Zr'lX^^^^^^ " 

With th:t;t%tL\t:S^^^^ beginning 

of the response in terms of the ufiTidanM^^^ destination 

number points to the column in Table D In ^mS "^^^ " 

bfer is stored in the first row. ^ channel num- 

^* DBP-to-stud^nts. messagp h^■r,A^ ru^ 

is commu nicated the -response, based uDof 't^ ^PProprx..te message switcher 
Tagging data, extracted from Table D ?s ^^^r. ""'^ ^^^^^ ^- ' 

output file, followed by the -ss\\°'f rl^h^"DBF?"^^:Lg^^^^1.\" 
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eludes: message type code, length of remaining data and message, 
number of destinations, destination codes, origin code, and time since 
• taidnight. The response itself follows. When all messages have been 
written out, control goes back to Step 1 , the student- to-DBP message - 
handler. 

i 

^' DBF messa ge flow "controller . This job operates independently 
and functions exclusively to receive responses from and' send commands 
to the data base processor (DBF). Commands for the DBF are recognized 
-by a bit on in common memory. When -it is recognized, the command -Is 
read off the file, using the length and address data directory, and - 
sent to the DBP^ through the external communications buffer. • This 
function alternates with the response. receiving function. Here the 
content of the communications buffer is transferred, to the response 
file, where, it will be read by the concentrator. Again, a bit is 
turned on by the message ffow controller to indicate to the concentra- 
tor, that a response is waiting. If this job proves to be "a bottleneck, 
\ ''^^^ into, two semi-indepehdenf jobs: one for commands to 

the DBF and one for responses from the DBF. 

Experiments will be performed to dete^ne the most efficient 
way of sending commands to the DBF -- that is, whether to send one 

arSforJJh™ """f^ messages stacked together, and if stacked, 

an^algorithm for getting the quickest respbnse out of the stack each 

•The concentrator also acts as a filter ^or various ^housekeeoinz 
^, and FILE. When a student sends any of these commands, he will 

f^e acti^eJS'be^S::?! • the DBF h.,t from the .c:^rL..L 

will be totally -simulated since ac tually conveying any^ of 
these commands to the real DBF could drastically affect the LSvitv 

witf a fUe^nlSbe'r a1 '^^^ ™d 

with a file number as an'^argument, the contents of Table C and Table 

\oi:'?:i::^::^rL^^ tranL^^teit^k 

u cne scuaent at his terminal over the communications 'n,^.,^ 
standard responses can be read from a text^S associated ^ItJ^th! ' 
concentrator, or they can be part of the prog^L's code 

have of thf L\c^^Sat^^'L^^^^ ^''^^\ ^% ""^^^^ 

tached to message switcher chanLi 2 ^bi'J^^ S '4 slSa h 

in Student 4's c^L^ds indiLt: ^hat 1^'?""^ ^^^^ 
to receive his responses T^fLfi I, ^'^^ proctor are 

nis responses. .The following commands are issued: 
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ouuaen u 


Connnand ^ 


Sequence 


Id no. 


Issued 


1, 


3 


c 

BEGIN 7 


2 " 


4 


BEGIN 11 


3 


3 


SEI.ECT (arg) 


4 


3 • 


EXPAND (arg) 

t 


5 


'4 


' SKLECT (arg) 


6 


3 


LIMIT, (arg)' . 


7 


4 


EXPAND (arg) 


8 


4 


SELkCT (arg) 


9 


3 


SELECT (arg) 


10 


3 


COMBINE (arg) 



r.. A ? ^f^ u^d^tBd after each of the above coinmands are 

read m. After all these coinmands are sent to the DBF throuzh the 
concentrator, the tables appear as follows: / 



Table A 

3 " 

4 

0 
-^3 

3 

0 
-4 

0 
-3 

b 

4 
-4 

0 
-3 
-3 



Table B 
1 
1 
2 
2 
3 
4 



Table C 
(1) (2) (3) (4) 



(1) 
(2) 
(3) 
(4) 
(5) 
(6) 



1 
3 
5 
6 



2 

■4 



Table D 
(1) (2) (3) (4) 



2 
7 
4 
1 
3 



4 

11 
2 
2 
4 
1 
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Table A lists the sequence of inputs by, student identification 
number. Since the students, are on different data base files, 7 and " 
11, the .concentrator must. issue a .FILE command whenever the input . 
changes from Student 3 to Student 4 or vice .versa. This concentrator- ' 
issued command is indicated by a '0' ir Table A. All commands which 
result in the creation of a set (SELECT, LIMIT, or COHBINE) cause • 
the sign of the student identification number to be changed ta a nega- 
tive. 

Table B lists the sequence of student set -numbers. In the third 
command. Student 3 creates his set 1; in the fifth 'command-. Student 4 * 
creates his set 1; ^.n the sixth command. Student 3 creates his set »2; etc, 

,Thd columns in Table C correspond to the student identification numbers. 
The first, third, fifth, an4 sixth sets created by the DBF are the 
first four sets created by Student 3; TKe second and fourth sets 
created by the DBF are the first two sets created b'y Student 4. 

* * * 

Table 'D accumulated miscellaneous data. The^columns correspond 
to the student identification numbers/ The first row indicates that 
DBF responses destined for Student 3 are transmitted over channel 2, ^ 
and DBF responses destined for Student" 4 are tran^itted over channel 
s;„H!^^ f f row indicates that Student 3 is searching file 7 while 
Student 4 is searching file 11. 

The third .row of Table D indicates that Student 3 has created " 

^^^L^^r."*^^^ ^•'"^^"'^ "^-^^^ "^^'^^'^ ^'^ ^^'^^^ The fourth row indi- ' 
• cates that responses to Student 3's commands have only one destination 

Sfth ITJ'Ttt ''fT ' ' '^'^"'^ '"^^ destination^! Se • 

to Mn. n indicate that responses for Student ? are sent'' 

proci^rrcodrd '^"'^"^ ' ^° the 

^♦^♦^ Billing an d costing; programs 

lnH^,,J5^ purposes of these programs is to determine what to charge 
Individuals and groups for their its<» nf rrnA Z , cnarge 

Clonal program he want's Th» „^ f s'"iJeiit chooses the Instruct 
tions £?las' „111 a^ aLgnerbf he\?m °' ^"fS^-S-P"— 'i"- 
instructional program and to1n°ta1lab1a'L^\%VreiSe" J'^rone""'' 
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with two channels available. 

The names of these files will be recorded on a file conmonly 
accessible to all the message switchers, as has been discussed in 
Section 3.6.1. When a strident logs out, the billing program will 
. record the time and will remove the names of logging-commiinications 
files from the commonly accessible file. This program .will also re- 
cord when the DBF is loggad in, when it is logged out, and the dura- 
tion of usage of- each data base. The actual logging in and logging" 
out will be -done by the proctor, while the changing of data bases 
is done by the concentrator. / 

At first, charges will be based directly on connect time as a 
function of 0 which PDP-11 program and which Lockheed data base is 
being used. A rate table will be set up for all the tutorial, exer- 
cise, and assistant programs. Also, a table of current' Lofckheed 
f.^lt^ u "''^ ^^^^ ^^^^ allowed by the IIDA system will be stored. 
With the exercise and assistant programs, at least the data base 
connect time rates will be charged. 

Blll/in? f ^r^f °" ^^'^^^ individual or group basis. 

Itll 7f i ^^""^ ^""^ ""^^ °^ "'^^ "^^8^5 the type' and 

Sese Mltf the extension of each entry; and the total charge. 

These bills will probably begin as paper entries bu.t will become real 
MliLr ?11 ^""""^ '"'^y operational. At that time! 

eStrience ihi^ w??r' ' represent valuable data and 

rTJs'Z: Si^Lfb^Virireai^L-^^ 

Costs to the IIDA system after it becomes fully operational win 

head. • rent, electricity, and other administrative over- 

A cost sinmnary report will >ip n-rQT,^>.«-4 ~ , * 

per hour will vary cons iderSlv ^V^^?^'^^'^ °" ^ ^^^8"!" basis.' Cost 
the cost can be Snerated S ^' t "Pf ^.on the distribution of 
tern's inco": o^er'^^L^'l^^L" ^'^^ 
.of the number of users and the t^/Tf' e ^^eak-even point in terms 
usage^ and alternative projection?^ P^of itabS ^ ^1="^^^^^ .-tual 
alned and a sliding raL scheduJ^can bf eftabli'sJed L"^'' 
usage during less profitable periods? "^^^^^^^hed to attract more 

3.7 Data Management Subsystem 
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(such as disk), retrieving it on demand, making changas in content, 
and performing sear.ches given appropriate specifications. In the con 
text of IIDA, the data management subsystem is a'^set of programs 
that suppott mother progr is by providing some of these fa'cilities. 
Reliance will be made to a great extent on the data management: facili 
ties provided with the cftmputer operating system and some additional 
programs will be added to these. We have the advantage over many 
data management system users that, however complex the files in the 
-DBF may. be, -the files withib the instructional and communications pro 
cessors are relatively small and simple in structure. 

^The IIDA data base consists of the communications log. the stu- 
dent performance history files and the text file. 

\ 

3.7.1 File organization 

' for f P"f°™^"'^^ history files aVe structured to allow ' 

for efficient data storage and' dynamic growth. (See 3.5.1 through 
VJ^'rL Z ^^^^^Y^ descriptions of sub-files.) For reasons related 
for cL^^Trf^^r °f indeterminate number of entries . 

for certain fields, it is important to relieve the applications.oro- 
grammer from the task o^ unpacking elements from these files e';' 
time a dataj/alue is required. " 

•* ' 

callp?^i,r°^''^ "^^""^ r"^^" '^^^ unpacking of the history files is 
caUed whenever a student entry is a log on, request for /HET-J 
parsed command, or loe faff - Tho <:-ti«, a V '•equesc ror /HELP, 

update routine;. Whe^ty:,f^LsfLti:ii°"^^%^ '"'^'"^ "^'^'""^ 
-'the enUrp n^rf^^ 7^^^ initiating conditions are met, 

..anlpuUting .ntrles in mulU^incSrriig fSlds'' 

by appucauorj^o'gja^^^' IfS then encoded and accessme 

the history file I3 S^;aliH ?^ condition is a valid coMnand, 

..ese_..sponse.pa.serSt"e'5:-\rS^^^^^^^ 

The student performance data ha<»P «-tii u 
manner, using (an indexed sequen^Lj f ilf o^.^'-, "''^^"^"^ sequential 
this file will' always be calH !' "'^Sanlzation. Records of 

trieving the data for a narScul!^ ^^^^ ^^en re- ■ 

for.^ng an analysis a^LTIu s^de'^f ^^^^^^^^^ -hen per- 

Of the file. The fJ^^y be sear chef '^^'^ ""^^ 

key based upon tl:ne. tSs would e nab L'^h ? sequentially or, by 

time to be located by a singL coZt%o ^he laL'T ^''^^^ 

unudua CO the data manager program. 
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The text file Is ^Iso an index sequential file. It is called 
for use or for updating always by frame number with which enTnries 
are associated. 

> 

3.7.2 Report generation 

c f^^f the sLudent performance (discussed 

in sec ion 3.6 3). traffic analysis (3.6.2) and costing and billing 
(3.6.5) summaries. These reports are generated by a series of special 
ized programs which use the student history data bases and the communi 
"rtl T ]°l "^:J°'^/°^'^ces of input. The reports are produced at 
regular intervals and upon request. The requestor may select from 
a menu of input options and output formats. The report print files 
are organised for flexible display; summaries may be directed to 
the line printer, hard-copy terminal or CRT screen. " 

Student pe rfo:Wnce rep ori-. The statistics aoDearln.? ^n M,-to 
'"^"'T f^-setul forlvaried applications. A standarS fo^^ft fo^ 
summarizing performanc^ wiU be- developed. Users of"he r^o^t will 
have some control over Vhich student records are included EurtM^ 
will be limited both by ^, the programming cost of ienerauf^tJon InJ 
privacy protectipn for students? ^ ^ ' generalization and 

(dlscSs%d"jf::cti':r3''7arSirfbf ^-^^^"^^^-^'-P-l^ing routine 
student data basef The dintl^L of ^ T'"""^^ attaches. the required 
and written 'o^l^l^r^flX.l^^^^^^^^ — 
cumulated for later posting of sLmary data |' " ' 

providr:orm^Xt,°f,:?Lt?:s '^s?°"? ^f"'^ i^^^ ^^^^^ - ^ 
quencies ar.^tabul^S' :ndt ns^'ompuS" ^^rrSultT'"^'' 
in matrices of varying deereps of nJn^i 7' . results^ are presented 
selected. • ^ ^ <i^SVBBS of comjJlexity depending on the options 

-put; atigSi?fj^^ J^:ff If x^^r '""^''^'^ • 

3.6.2 describes the data tabulated ^^^^^^^f '^^PO"- Section 
the report generatorlor tMs infor^ffr%' of 
■and format the printed output : ^"'^^ '^^^ P^i"'^ file 

P-^-; aII45^^?ffci^^ costing progr^s 

section 3.6.5 for details. JhrrfpL nro °^ '^"P""^' ^ee 

the billing/costing files anf LSq^L^^'^da.f L'tputf"^' 

3.7.3 Ad hoc q uery \ 
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There will not be svch a facility in IIDA,^ on the grounds of programming 
cost. Analyses of performance, communications activity and cost will 
be programmed as standard reports. These may, of course, be rspro- ^ 
grammed, if necessary. As new requirements arise, they may be pro- 
grammed for analysis. But therte will not be a program that permits 
IIDA users to define a new information requirement and have the re- 
sulting file search and formatting be done immediately. 

3.7.4 Text flies ... , - 

A text file makes It possible to completely segregate the text 
used In each of the, programs from the programs themselves. Changing 
and adding to the text become simple operations for the person who 
is vnriting the course. All textual entries from this file can be bti- 
coded into a few bytes on the logging file (see Section 3.6.2). 

\ 

.Each program in the IIDA system (the tutorials, the exercises^ 
and th^ diagnostics) will have its" own text file. For purposes of 
the text files, these programs can be described in terms of a number 
of frames, each of which is subdivided into a number of text-groups . 
' A-f ralne is a cpqf eptually or structurally related unit of instructions 
or discussions. A text-group is any contiguous block of text within 
the frame . . 

- \«"^""tine in each program will call for a single text-group 
from the file and- print it at the terminal. This subroutine is also 
responsible for making a notation on the logging file that the text- 
group was printed. Two ^arame.ters are set -before calling the sub- 
routine. These are the frame number and the text-group code. The 
frame number may be set at the top of the frame so that only the 
text-group code need be set before everj- text-file subroutihe call. 

that S """'^ ""'dieted to only information 

fSl ^ILl ' "^^ "^^^^ "'^h" subroutfnes to 

fill arrays which represent possible anticipated answers to a tutorial- 
question. Branching to alternative routes -through the tutorLr^^f 

ent'r:' sJ^iL':/"'"'^? '° '""^ ^^"^ ^^'^ which'matches" he's%'uSL 
arS'on iJe ^S'fn! ^sspellings may be set up in an' . 

l^t rte^t^g^oT.^^^^^^^^^^^ 

handw\5^ir ^^^"'^i^Jf "i'^h programs in the IIDA system wiU be 
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Two data items that^pertain to the entire file are the frame 
count and the wiped-out byte count. The. former^ contains the highest 
frame number for vrtiich there is data on the file. ^The latter contains 
the count of bytes on the file which have been discarded because of 
adjustments made to text-groups* The use of the wiped-out byte count will 
be discussed below under "reorganize." 

The frame directory, also on the first block of the file, con- 
sists of pairs ^of numbers corresponding to, i.e., indexed by, the se- 
quence of frames. Th4 first element of the pair indicates the block 
number where that frame's text-group directory begins. The secqnd 
"element indicates the address on thafblock where the text-group direc- 
tory begins. Given/the frame number, the address of any pair in the 
frame 'directory cail be. calculated to the byte by : d+ 4*(f - 1), 
where d is the numtter of bytes devoted to. data which applies to the en- 
tire file, found at the beginning of the first block, and f is the 
frame number. The number ia'the formula derives from the four 
bytes Allocated to each frame for the block and address data. 

The text-group directory consists of pairs of numbers indexed 
by the sequ-ence of text-groups. The first element of the pair indi- 
cates the block on which the text of the group -begins. The second 
element indicates the address on., that block on which the text of the 
group begins. The address of aay pair of numbers in the text-group 
directory can be calculated by .the formula: a + 4*(g - 1), where - 
a is the address? of the text-^roup directory and g is the 'text- 
group code. ^ If this code expeedr the limit of 512 bytes per blocks 
the next block is read In and 512 is subtracted from the address 
calculated above, yielding the address on the next block. Twenty ■ 
bytes are allocated at the end of each frame's texc-group 'directory 
for up to five text-groups which may be added later to the fraie. 

.r.A ^^""^ prefaced by two numbers: the sentence, count ' 

and the byte count. Forty bytes are allocated at the end of each 
text-group for later expansion of the text. The byte count includes 

is no3r J^""' ^'^'^ the next. This 

is normal It, would be exceptional for a group of text to end with ' , 
the end of the block. Lines or "sentences" are delimited by. the two 

lZl°llL ^'^^ ^^^^ ' ^^^> • ■ When theLe har 

been encountered the numbeir of times indicated by the senten'ce count 

blocT^i%\°' text-group has been completed.^ Overflow IrL one ' ~ • 
by The LnStVS' Irl J'""'"' 1^ maintaining a pointer iLremerited ' 
'is called fn? ^^"tence. When this exceeds 512. fc^^ next l?lock ' 

_ -Four programs are available for maintaining the text 4iles 
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1. Create , This program allows whole frames and text-groups to 
be added to each of the files • The appropriate f ile. ia first indi- 
cated by the course writer and then established by the program. Then 
a frame number is entered and a text-group code is entered ♦ Alter- 
nately, frame number or text-group' code may be automatically assigned 
as an option. ,Line or "sentence" numbers will print on the left. A 
text-group is entered line by line by the course writer. The entire 
text-group is stored in cor^ and the total length is calculated. 

The text-group directory is pointed' out by 'the frame directory. For 
a new frame, a frame directory entry will be established. The' next 
possible block and address will be picked up f ro» the 'file data area ' 
on the first block. This information will be written in the text- 
group directory corresponding to the frame. The text itselfr will he • 
written at the end of '^the ^ilp. Overflow from one block ^to the next 
. will b^ managed au:tomatically. Any attempt to write a text-grpup 
to an old frame whose text-group directory expansion area is full will 
.be met with a message that the reorganization, program should be run. 
This message will app^r before ar^y text is entered for the group. 

2. Add. This program allows sentences to be addpd anywhere into' 
already existing text^groups. Thef-course writer indicates the file^. 
with which he is working. The program establishes; the appropriate- 
file. The course writer enters the frame number; "the text-group 
code, and the line number following which additions are to be made. 
iwJJfn w!^ ^^"^^ immediately prior to the addition are 
printea with their l^ne numbers for verification and reference. Addi- 

^7S.^^^^ ''"J^r^A ^^"^^ entered. Line numbers are printed for 
each line added. An' opt,ion set at the beginning of the program will 

liiruSbeSr^"^"^ ^^""^ text-group to. print, with their new 

^ All added line^ and all lines following the addition are stored 
Sriri^Ittn\°"; ''""^ completed, thJs c^r^-s^red 

for if T ? text-group area if there is enough room 

^s ri^itSntt'thi've^ "T^) T"^ °' -^^^^ s'^^P 

IS rewritten ^t the very end of the text file. The appropriate corre- 
sponding changes are made to the text-group directorv Th! ! \ p 
bytes thus freed are added to th. ^iped-"? tylT °7ni °' 

text-a;ou?^: sentences to be deleted from any " 

text group. The course writer Indicates which file he is workine Zilh 
The program establishes this file. The ^ourse writer enters =th» 
d^Wer f' ^^^J:f-P -°<^--^ and tt line^^^bers o be 

With thJr line number: r::^;.. -r e'"::,,^\^LttTirhe^?r'' 
he want« these lines deleted. Upon verification ttlL I 
parameter at the top of the text-eroun 7« .^f i'. . ^^"''^ """'^ 
lowing those deleted are moved un IT .^^^^^^^^ the sentences fol- 
If the entire text!gro;p TdeleLf rhrblf ^ ' "J''^' —-deleted.. 

the text-group director^ are set to 'zero" S ■ 

y die. sec to zero. The number of bytes thus 
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freed are added to the wlped-out byte coiint. 

4. ■ Change, There is no prpgram which directly changes lines of- • 
text. Rather, changes are accomplished by deleting Unes and then 
adding lines with those programs designed to do these functions, 
discussed above.- \ " 

?' Reorganize. This program adjusts data on the file for optimal 
'disk utiUzation. If a text-group directory is full and more text- 
groupa are to be added to the frame, or if the count of wlped-out 
bytes exceeds a set percentage of total bytes in the file, a recom- 
mendation to run the.reorganizatiion program will be issued under 
program control in one of the. other maintenance programs." Reorganiza- 
tion essentially involves copying the entire text file over into 

\another file,' by changing any directory parameters necessary in order 
;to ..restore 20 bytes to each text-group directory expaniiion area, -and 
to restore 40 bytes to each^ text-group area. All areas wiped-out 
in the old. text file will be eliminated from the new text fil^. -A 
first pass will determine, how many b J ocka "should be allocated to the ' 
text-group directories. Acsecond pass will perform thrf actual copying 

■ tL T ^^^^ '^ill be scored in core on the fiSt - 

secdnrpaS^"^"" '° ^"""^ text- group directories on , the 

3.8 Systems and" Programming Support 

TT^^* briefly describe the mechanics of linking the - 

IIDA computer with thfe data base processor. This is all done with 
standard commercial hardware and sof<tware and is. made simpler by the'" 
fact that the IIDA computer, appears ,to the DBF as just another terminal. 
No messages are exchanged between th^se computets that are different 
froA messages between the DBF and any user terminal.'' Similarly,' the 
DBF looks to Che IIDA computer -as a terminal, although IIDA expects 
different. kinds of messages from the DBF than from a .student terminal. 

Both the establishment of the computer-computer link and writing 
its, associated programs, and the restart-recovery program are tasks 
during the implementation phase. In addition to these, which have 
specific designs and goals, there is a third general support task which 
is to provide operating systems consultation to project programmers. 
.One member of the staff must become expert in the operating, system used 
and be in a position to advise the others, or to get advice' from the 
manuracturer, on detailed use of the operating system and compiler. 

3.8.1 , Computer-t:6-Computer Linkaf^ e t 

. The IIDA PDPll/34 and the DIALOG data base processor will be inter- 
connected via .a switched network (either TYMNET or TELENET) . Functionally 
the .network and host processors and the IIDA computer will yiew each 
other as ASCII terminals operating in a full duplex mode with odd parity. 
Communications -will otcur at a transmission rate of 120 cps. 

^. ■•' 113 ■ ' 



- 103 - 



The interconnection of the two computers will occur upon user 
demand that is, when the first user in a day, or; the first in any 
continuous session, issues a command to the DIALOG data base. (Only 
this user would ^experience a delay while the link is being established.) 
The following steps would be followed *to establish the interface. 

I . * 

The proctor, as a privileged user, will run various utility pro- " 
grams under the RSTS/E operating system to effect communications. A 
port wlil have, been^ designated solely for communications with the 
,data base processor. This surrogate terminal will be configured during 
the system initialization phase causing the operating system to recognize 
pre-specified characteristics (suoh as baud rate) every t;ime connection , 
is pade on that line) 

c> j 

The prc^;tor will then manually dial the switched network Via the 
local service network over ordinary telephone line's. If the response 
is a busy signal, ^^a rare occurrence, the proctor will send a message , 
to the user infotming^ him that the system is unavailable and advising 
him to logoff* This and other proctor messages will b^ .Ce^cprded onN 
text file and transmitted following proctor input of a' transmission 

code. . . ' ' 

«* 

A successful connection will be confirmed by an audible tone. 
Communication thus established, the proctor will place the receiver in 
the acoustic coupler. He will then process a file which' calls for 
a series of commands that will carry out LOGIN procedures on behalf of 
the "pseudo'' terminal. Other commands will call, rtin, and detach the 
message flov/ controller routine. An example of establishing such a 
command file, -referred to as a control file by DEC, follows,: 

Command A ction 

LOGIN. KBl: 1,5 RTST/E will login terminal 1 

under account number 1,5. The 
^ operating will look up"^ the cor-^ 

responding password. r 
FORCE KBl: RUN $TTYSET Will process the command to set 

terminal characteristics for 

terminal 1 

FORCE KBl: KBl: Will cause OS to recognize that 

terminal characteristics are 
being set for keiyboard 1 
FORCE KBl : 71*0508 ^ RSTS/E will recognize "pseudo" 

terminal as a DECSCOPE for example • 
FORCE KBl: EXIT v . SignifieiS the end of the TTYSET 

routine ^ > * 

FORCE KBO: RUN $MSGCTL Run message flow controller from 

^ . ^ t}ie console 

FORCE KBO: /HE - Detach message flow controller 

■ . , ^ program 

' END ' ' End -of control file 
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• «n A<i'^r ^Y^u^^J''^? from 'the network computer will be converted to 
an ASCII code through the modem, transmitted . through the multiplexer, 
and recorded in the incoming message file. A bit in common memory is 
turned an to indicate that a^DBP response is awaiting action. The 
message flow controller reoo,^izes this kt as an unsolicited response 
indicating that the network machine is awaiting login data. The front 

'^^^^ ""^^ mamier>of a "network access machine 

by seiiding appropriate access protocol -^messa^es which will' establish 
network and host computer communications. T^o sets of messages will be 

depending Sh whether connection is attempted 
for or^TELENET. (A minimal number of conpands are requSed 

for access; therefore they will be stored as literals within the context 
and Xarr rather than as separate files to be opened 

The^a are examples of network access messages to be stored and 
transmitted: 

"^^^^^T ^ . TELENET 

• terminal id = (CR) (CR) 

(CR) ' terminal id 

P^O(i C 415^20 (CR) 

Dialog password • (CR) Dialog password (CR) 

• as they"arr"ri^1h!°?'^":^"^' ""'^'^^ °^ ^^'^^ ^^^^ responses 
as tney arrive in the incoming message file to ascertain whether the 

computers have been successfully linked., It is possiSe thkt eJtJer 

im^be^LMn?" "^"^^ '"^^ connection. ?Je pro-am 

J^Ll "Etching messages for anticipated responses.. HaviAg detected 
an anexpected response, the program will .send thut mess^S ?o the 
tSfnio^ be examio^edby the -proctor. Proctor input Lfs^persede 

loltlolZT^^lllZlT^^^^^ =«y inform the message 

controller to retransmit all- or part of the access messages, to wait 
E minutes and retry, or to discontinue attempts. He may also entS 
Se n^c?^rT?"^rw'"^'"^'''°"- " '"^^ connection can^rbe^e. 
"dowS"? ' ' ^ '^^^ network/DBP is temporarily 

contrS^fof tr^n3^M°"''^^ ^^"^^^^d DIALOG is online. 

tSin resSe St?^. ^ capturing data from the DBP wUl 

then reside with the communication processor's applications programs. 

3.8.2 Restart and Recovery • 

PnPll/i^^''^° ''"f^' P""^'' brown-out or other malfunct^'ion involving the 
PpPll/34 may result in disruption of student interaction with IIDA 
programs and the data base processor. The .RSTS/E opetati^ system will 

^Ss L fccornnlLEL J can theoretically be easily effected. . 

5 ""^^ operating system performing several tasks 

contSs"or^ ^ "'^^"^^^^ ^'^'^^•^^ '^^"^^"^ fil-- and^saving ' 
the contents of core memory and .the general registers. 

• - lie 



However, whatever data in the user's buffer that have 'not been 
put out to disk' will not ,be preserved. There are likely to be, at •• 
any given moment, ^several, simultaneous users' whose' history files will 
be in various stages, of update and analysis, the RSTS/E automatic 
restart facility cannot guarantee that a user entry has been.read from 
the input buffer and processed. ' ' 

Therefore, a specialized IIDA restar4program will be run upon 
recovery. The student will be advised that there will be a slight 
delay before he is once again conversant with IIDA. 

* 

. The restart program will then reissue' the sequence of valid 
user commands from the first to the latest completely updated item.' 
-The lo,g file fill provide the necessary , data for resubmission. Com- 
mands will be stacked and sent to the ^data base processor. Data base 
• responses will not be transmitted to the user but utilized to restore 
the history files. 'Data from the traffic log .such .as ^relevance ratings 
will be maintained. Times accumulated, during the restart rtin will not 
be recorded since they are not meaningful .-for performance, analysis 
purposes; rather the original times will be transferred from the I02, " 
file to the history file. , , ; 

To aid in restarting. after failure, certain procedures oust be 
followed during update routines. A bit w^.11 be set wheq. every update 
to the history file has been completed in response to a command.- By 
examinlijg this bit the restart prd^am will know and can tell the 
student whether a command needs to be reissued. 

* 

Thresholds may be crossed while re-processing commands. The 
restart program .will respond by bypassing PAR calls and resetting 
threshold -counters. In effect, the routine will do the bookkeeping 
without actually performing the analysis. 

' . - 

A crash during a PAR run can be detected by examining the update - 
completion flag. Here the user must reenter the command since history 
files may be in a p,artially updated state. Counters will be decremented 
accordingly. 

At the conclusiem of thXrestart run the student wilj be informed 
of the -final command (fully processed. He may continue the search from 
that point. 
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. :^4. IMPLEMENTATION PLANNING AND EVALtJATION 

4.1 Overall Utilization Plan ' / 

It is Important^ to distinguish between tt?o types of planning 
for promoting the use of IIDA: (1) fa'cilitating and promoting 
initial use in sitsuations where th^ primary concerns are to prove 
the concepts, to obtain, the ^feedback necessary to improve' the 
design and operation of the syst^^ and to gather support in the 
user community; and (2) marketing of the system where the ob- 
jective^ is to m Imize use a^d to' achieve enough Yncome to make 
a profit, support future research, or at least sustain the system 
on a sfelf -paying basis ♦ Our present concentration has been on 
ways of introducing the .system into a user environment where 
it will be appreciated^ from which we can, learn, and which can 
lead to support wher^/lncome-oriented ^larketing begins in the 
future. This is^ necessary in order .to refine the systm and 
develop a knowledgable and, loyal user group. li^ the system is 
to be eventually sold on a broad scale it- will be sold as a 
result' of proven performance. This proof is best manifested 
as a result of gradual, steady, planned growtji. 

Inherent, in the need for the feedback mentioned in the 
previous paragraph is the recognition that IIDA^is still an " 
experimental concept. AlthougK"" IIDA^s designers are convinced 
of its feas.ibility and utility they recognize that successful 
commercial exploitation of new products often requires ex- 
perimenting with- the ways in which users will accept th6 idea 
and posslbily revising the original ideas. 

While our major focus has been bn planning ways to get 
started with initial use we have not neglected planning for 
the marketing of the systen on a cost-effective basis. Our 
approach here has been to look for a situation which, following 
a dif fusion~of-innovation model, w?.ll allow us' to' select an 
initial' user group which will not only serve to provide useful 
feedback on the •design of the system, but^which will also 
provide us with an entry point into a broader user group. 
Thus, the-inltial user ferdup should also serve as a means of 
educating a broader potential user group about the system * 
and as a means of diffusing knowledge about IIDA. 



4.2 Possible Initial User Groups 

When IIDA was first conceived, we assumed that usex ^ would 
be almost^ exclusively industrial. Our rationale *was based upon 
two factors: (1) the enormity of the potential user group— 
^somewhere between one and three million '^scientists" in the, • 
country; and C2) the then-assumed post. We' felt that only in- ' 
dustrial users would be able to afford the training cost, 
that those. who could or would go to classroom training should 
do so, and that this form woul^d be more cost-effective. 



While ultimately this potential user group should make 
up* a fairly high percentage of the users of IIDA, the^,e are 
several factors \ihich suggested to us a different group of 
potential users should be targeted as our initial user group. 
First, Arthur filias at BIOSIS, and John Creps at' Engineering 
Index have both suggested that their experiences indicate 
^.that industrial scientists are less receptive to innovative 
information ijrograms than are faculty and student groups. 

A second reason for change is that the cost of data-base 
search service is coming down as a result of competition, in» 
creased markets and improved technology. 

A third reason is that- our technical design work indicates 
that the concentrator will enable more than one user to share * 
a line to a data-base computer. We hope to be able to negotiat 
'lower rates ^fpr .data base services when working* through a" 
Concentrator. \ * 

Jourth, student users are also more attractive as 'an 
initial user group because, as they, move into industry, if 
they are satisfied users as students they will be likely to 
continue to use the system in an industrial setting^ and to 
promote its use by colleagues who. have not used it. 

Finally, although it Is nearly universal in universi- 
ties today that conventional Computational services are pro- 
vided by a central service in ample quantities to academic 
users, search services are rarely made available in this 
way. Currently a break in this pattern is beings negotiated 
an our own universit^^ and we believe that eventually, biblio- 
graphic search services are going to be made available to 
academic departments, for both faculty and student use. In 
addition, we believe ^that this practice will be*adopted else- 
v^ere because the service Is needed and will be eventually 
recognized as necessary. The only question is when it will 
happen. , 

As a result of these changes and a positive attitude on 
the part of educators,^ we now feel that, academic usiers should 
"be the first target of our attempts to test the system. An 
additional factor which suggests that ^academic users should be 
the initial target for preliminary ,use of IIDA is the existence 
of the Experimental Partnership for the Reorientation of Teach- 
ing (XPRT) program, a consortium of five engineering colleges, 
which is headquartered at Drexel and funded by NSF. The de- 
gree of interest expressed by the XPRT people at Drexel leads 
us to believe that this -group on this campus provides an ideal* 
initial user gtou^'. ^ ' ' 

When Integrated into the ongoing activities of the Stu- 
dents and faculty involved in the XPRT program, the use of 
IIDA by this group will provide us with the needed feedback 
in the design of the system. In addition it provides a 
natural, mechanism for educating potential, users abo.ut the 
•system and for diffusing knowledge about IIDA. Expanding - 
the use of IIDA into other aspects of the engineering curri- 
culum at Drexel and expanding the use of IIDA into thfe entire 
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consortium of schools represented in the XPRT program would 
provide a widening o5 operational experience ank act as the 
forerunner of a full-scale marketing program- This- approach 
also allows for the possibility of the diffusion of IIDA use 
into industry. This could occur not only through act^ive 
marketing, but also through. the diffusion of satisfied users 
into various industrial activities. 

'4.2.1 Utilization at Drexel 

The initial primary user group will be senior level and' 
graduate level students in the various engineering curricula 
at Drexel. For this user group we will use the Engineering 
Index's data base Compendex. Entry into this user group will 
be gained in two ways: (1) contacts established with the faculty 
involved in the XPRT program; and (2) courses in technical 
writing fur engineering students. Initial users will be 
approached via XPRT since this group on the Drexel compus 
provides a structure and set of on-going activities into which 
IIDA can be integrated with maximum impact and minimum its- 
ruption. The faculty selected for participation through this 
route will be those involved with the XPRT project and hence 
committed to innovative teaching methods. In addition the 
XPRT organization will provide a forum for discussion of 
how well the system works and how it can best ba integrated 
into the oVerall curriculum. Discussion so far indicates that 
data base searching should probably not become a separate 
course. Rather it should be offered as a sewtce to Engineer- 
ing students who would use it in' connection with* senior design 
projects or. graduate theses. Students would be given assign- 
ments involving liibliographic searching and given the. option to 
do searching through IIDA or. complet;ely through the library. 

The second ' entry 'point involves courses in technical 
writing.-; A recent revision in Drexel*s engineering curricula ' 
is the inclusion of a required course in technical writing 
for all engineering students. Discussion with the faculty 
presently on campus who are scheduled to teach the technical 
writing courses indicates enthusiasm^ for the lIDA system on 
their part and a willingness to work with us in incorporating 
IIDA into their courses as a service to the engineering students 
in writing term papers. We also have reason to believe that 
some of the engineering faculty not associated with the XPRT 
program would be receptive to ^this approach. 

It will be possible to arrange workshops for faculty to 
provide a background in the project to a broader range than 
just those faculty who are afctive in initial participation. 
These workshops will provide feedback from IIDA users ^nd 
help to build an extended ne'twork of users and potential 
users at Drexel who have shared the experience of IIDA use. 
Case-histories of users can be collected and written for 
dissemination in the -workshops^, describing how others with 
similar backgrounds and needs have approached and used the 
system. These case histories can also be used in contacting 
students and fs(culty who are unable to participate in workshops. 
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4»v2 . 2 Extension to other schools 

The next step involves extending IIDA usage to the en- 
gineering classes and faculties in the XPRT member universi- 
ties, where there is a need for bibliographic searching, using 
both workshops and case histories. The experience gained on 
the Drexel ' campus , and the already established liaison between, 
the XPRT participant campuses, should greatly facilitate this •'^ 
extension. Once IIDA is in use on the XPRT program campuses 
the next. step would be to extend IID4 usage to other si;hools 
in the mid-Atlantic region and then eiaer further. 

4.2.3, E xtension to industry , ' 

We also expect to be abl^"^ to, extend the usage of IIDA 
into industrial settings. Smith, Kl'ine and French Laborataries * - 
shave indicated a willingness to par^ticipate in tests of the 
jsystem and other local industrial users^may al^o be sought. 
Once the IIDA system had been put into use- at Drexel and the 
other XPRT consortium schools, an increasing number of engineer- 
ing graduates going into industry will havef had, we hope, 
positive experiences with IlfiA and its ca^^atilities. We 
expect that many of these users will .Be likely to want. to ' 
continue to use the system in an industrial setting. These 
satisfied users will provide a potential word-of -mouth ad- 
vertising network which should -prove to be particularly- 
valuable when potential industrial users are approached with 
information about the uses and benefits of the IIDA system. 
In addition the widening oi^ operational experience and the ^ 
marketing experience gained i-n extending the use of IIDA, - ^ 
throug^hout the XPRT consortium and other schools, taken in 
conjunction with the already proven performance of the system,* ' 
should prove invaluable in developing a marketing strategy 
for introducing IIDA ^ to users in industry. 

4.2.4 Pricing policy and support assumptions - 

The utilization and acceptability of IIDA depend upon 
price as well as on qt^lity of performance. Price, in turn, 
depends, upop costs, manner of distriSuting costs among users, 
financial support from NSF or other sources, and volume of use. 
In thi's'section daspuss,the variables affecting .cost to 
the" user.., A specific pricing plan will be developed';in the next 
.development phase. ' ' 

Costs . Costs may be divided in1:d two broad categories: 
pre-operation or development costs, and operatin^g costs. In 
ajl' planning for a pricing policy, we are assuming that ic 
x^ll not be necessary to recover NSF-granted development costs 
out of operating income. • ' 

' The ' operational co.sts exclusive of overhead are: 

Computer (The IIDA minicomputer): lease or^ purchase, 
' maintenance and spare parts if not leased 

Programming support for the IIDA computer: maintenance 
"programming and system modification ' \ 
i Tutorial and analytic support: the continued analysis 

of performance and improvement in design of the system 
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Evaluation: continued evaluation of performance 
Communications: communication links between users' 
and the'IIDA computer and between the Xatter and thi 
search service* ^ 

Search ^service: the costs of using tha data base 
processor 

2. Income sour&^es * « The following are the sources of "income: 

Student use fees; paid by themselves i or. their employers 
or educational institutions, " ' - >' 

NSF subsidy, espeqially to assist in establishing the 
IIDA service and- supporting it to the point where it is 
viable* <, ^ 

Vendor support: largely in 'kind, this can be realized 
through nq-cost assistance in-development and operation 
and, possibly, .in reduced costs of operation for students 
.or when using a Concentrator. 

3. Volume of^ use * Factors affecting volume of use are: 

Venture capital* This system is not being developed as 
a profitmaking enterprise supported by venture capital*' 
There ire no funds for advertising and other r:>romotional 
activities except through normal professional publication^ 
and discussion channels* 

Developmental nat>tfre of the project: When first offered* 
to test users, the system will not be complete'^and fully 
operational* Initial users will be test subjects and 
this will clearly affect their willingness to pay for 
services* The more the product is proven and publicized 
in service, presumably the greater the willingness of 
users'^'to pay for it* 

Initial use will be restricted to one data bas3 and 
one search service* Expansion beyond that point will 
make the system more attractive to users but cost an 
added increment for each new data base or search service* 

University central support* We have indicated previously 
the importance of',*university policies on central provision 
of computer services for the entire campus* Thesis common 
today *f or computational use of computers and, in the 
typical situation, users feel they are adequately supported 
in the computer needs, which are not charged to college 
or departmental 'operating budgets. Search service has 
not reached this condition yet and, typically, is charged 
to operating budgets of .individual departments and colleges* 
This clearly reduced use of bibliographic search service, , 
whether IIDA is involved or not* As universities are 
brought to recognize this use of computers as equal in 
importance witli>i0thers, this situation will be alleviated 
and more funds will be available for bibliographic searching 

Laboratory fees* Many universities are|beginning to drop 
the concept of laboratory fees, in ef f ect \paying these out 
of overhead Miich is then distributed among all students* 
Thus, ixi some* -universities, of which Drexel^ is. an example, 
we do not have the option of charging a laboratory fee 
for use of the system, and jjthis any affecf level "of use* 
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4. Price considerations . Aside from the need, eventually, to 
break even within- all the considerations of cost, income and volume 
of* use described above, the following' are the major questions affect- 
ing the price tfhat should ^be- charged: 
\ University or ^'ployer support. Will 'the university or 

employer agree to buy IIDa sei^ices for 'its students/employ*les? 
Y NST suppprt.' Will NSF support III3A until it is able to 

become self •supporting? ^5 

K Vendor price policy. *To what extent will v&ndors of 
data bases and search services support IIDA through assistance 
in , consulting, promotion and" special usage rates? ' 
^ • . When should IIDA try to break ev^n f inarioial^ry? 

Should the operational IID support continued research and 
development? t 

^—-4.2.5 Specif ic goals to^ await more detailed study and' develop- 
\ ment'' during the: implementation phase . 

Curing the implementation phase of IIDA development there 
ate several goals which oust be accomplished before beginning 
initial tise bf the system. « Several of these are listed below: 
Survey of -engineering and other , .faculty wHo work with 
^ engineering ''students and who might use this system in their 
courses. • ' ♦ - * 

^ - Selection' of courses -and faculty to work wJ.th. 
^ ' ' - Joint planning with selec*:ed faculty for actual usa. 

, - -Develo'i)ment of a more detailed plan f ot. extending use 
beyond initial test group. 

-I Preparation of "''training materials for use in connection 
with workshops, conferences, etc*. aimed at encouraging' new 
users. * 

-^Development of- a mor^e detailed plan fox. formative 
' evaluation at. the end of initial use. 

- Development of a more detailed plar^^ for broadening 
the user* base beyond the initial user group. 
^ - Development of a more detailed plan for* summative 
evaluation of IIPA Impact on broad user group. 

4.3 Preliminary Testing 

It is our- intention to perform mechanical 'testing of the 
system at all 'stages of the development of the IIDA system. 
Some testing, of tue model system has been conducted during 
the design phase and this is discussed in Section 5. Another 
system test will' be conducted near the end of the implementation 
pnase as ♦:he computer programs are brought together for in- 
tegrated system testing. There aria two kinds of testing which 
need to be performed — mechanical testing or de-bugging, and 
system testing. 

4.3.1 Instructional programming testing 

The educational programming aspects of IIDA involve the 
logic of course design and the actual programming of the tutorial, 
exercise I' and assistance programs' into the language of the 
instructional processor. Puring the implementation phase of 
the IIDA project the detaile'd course design must examined 
for internal f.laws^ or inconsistencies and for contormance with 
the original design. These program? must then be pilot 

tested on a small group of naive* volunteers before attempting 
to use them with the initial users. 1 

X, h»j * 
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4.3.2 Computer programming 

The computer programming aspect of tue Implementation 
phase Involves systems programming (Installing and testing 
the operating system, data communications package§> If needed, 
and any special data communications programs written specifi- 
cally for this project), programming of message store and for- 
ward and concentrator functions^of the communications processor, 
programming of a billing algorithm (for allocating the cost of 
the computer system use among users, depending on actual 
facilities used) , and programming of diagnostic ,and student 
data base processors. Each of these systems must be carefully 
tested for mechanical flaws. Once again, the operation of at 
least some of these programs must also be pilot tested on a 
small group o'f naive volunteers before attempting to use them 
with the Initial users. 

'4.3.3 System testing 

Once each component of the educational and computer pro- 
gramming packages of IIDA has been de-bugged on an individual 
basis they must then be assembled and given a thorough system 
test. Once the mechanical aspects of sys.tem* testing have been 
completed the total system must be pilot tested on a 'small 
.group of naive volunteers before attempting to use it with' the 
Initlal iusers. ^ While pilot testing of the entire system must, 
await completion of the total package of programs we expect 
thatfye can and will be able to do some of the pilot testing, 
as indicated above, with naive 'users on some of the components 
of the full system prior to actual completion of the system. 
In addition some of the aspects of the full system can be 
completed in advance of 'the date the full system is ready by 
doing slmlulatlon testing with the model system completed 
during the first phase of this project. * 



4.4 Evaluation 

Although we will eventually want to submit the IIDA 
system to both formative and summatlve evaluation, it does^ 
not make sense to implement both types of evaluation at this 
time. While the system is still in its early stages of ^ 
development, formative evaluation aimed at monitoring the 
progress of system development and providing useful .feedback 
seems most appropriate. We do, of course, also want to plan 
for the summativ^. evaluation which should be performed at a 
later time. 

As with any evaluation process, one should start with 
the goals for the system at rhls stage of development. 
There ar^ basically two 4najor goals which the IIDA system is 
designed to accomplish: (1) To provide an effective method 
of teaching and assisting in interactive bibliographic 
searching; and (2) To construct a workable, technically 
sound computer based system which can be successfully marketeS. 
These are both long range goals wiiich cannot be evaluated at 
this time. These are the goals, which a summatlve evaluation 
must address. 
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The dependent variable:.^at this point was ''user satisfaction"; an 
alternative dependent variable is "user skill" in operating the system. 
Measuring , user skill in this t;ype of experimental system is* critical. 

The role played by IIDA begins to change as users become more 
familiar with the charabteristics of the system and the mechanics of 
searching. At some point the teaching role of IIDA will become sub- 
ordinate to the assistance role. As searches get fully underway IIDA' 
will be available for suggestions and help about how the system can be 
best used to accomplish search tasks. IIDA may be called upon to act as 
a t;ranslatqr of search strategy. Also should the searcher get into 
difficulties which we can identify in advance IIDA may suggest options 
to the user which have not yet been employed or exploited fully. Finally 
IIDA will continually monitor the performance of the user to see if he 
is having problems with the system, even volunteering help should the 
occasion warrant it. 

As these developments take place and as the project moves up the 
ladders outlined in Figures 4.1 and 4.2, the dependent variable xd.ll ^ 
change to impact upon the user. This impact measure will be described 
in Section' 4. 4. 4, on plans for evaluation. At this stage given the 
state of project development, we have been most concerned about operationa- 
lizing user satisfaction and skill. i 

In order to operationalize the concept of "user skill", one must 
reflect carefully on the dynamics of interactive learning through IIDA. 
This dynamic can be quantified (operationalized) tlirough several routes. 
One of these, for e:'*>mple, could be an analysis of the messages between 
IIDA and the user. Presumably the user*s commands, errors, and requests! 
for help zan be classified into several categories such. as the following: 
learning, procedural, requests for review, etc. While not specifically 
developed for this purpose, the ability to detect the skill or the user,' 
and his^ errors, is, in part, being developed in the diagnostic routines. 
Skill error ratings can be accumulated- and updated as a function of a 
number of variables (e.g., commands used, elapsed time, requests for 
information, pattern of commands, etc.). These indices can be used to 
describe the proficiency of the usex in several aspects or different 
phases of system usage. Not only do the diagnostics allow IIDA to adapt 
its behavior to the demonstrated skill of the, users, but they also will 
allow us to assess the development of learning. The adaptation of IIDA 
to the user is also a reflection of the user*s proficiency with the 
system. Over time we expect to test for the extend to which users take 
advantage of the ^ull capacity of the system. 

While the dynamics of learning about Interactive searching will be. 
complicated to study they should be systematically related to the evolutipn 
of the search and to the reactions of searchers to the system. Although 
wa will initially only be able to study u^ers who have done interactive 
searching on a limited and short term basis, future analysis of long- 
term usage could reveal whether the dynamics of learning continue as 
people routinely '*search with IIDA. 
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It shpuxd be clear jthat these two major goals -have a . ^ 
number of"* assumptions behind them vhich preclude evaluation 
now: (1) an 'effective teaching method must.be evaluated 
relative to other methods available to users and -potential, 
users; and (2) no comparable assistance system exists; (3) 
a workable, marketable system must also be evaluated relative 
to other bibliographic search systems. 

4.4»1 Evaluation at this Stage of^ Project Development 

At this stage of project development several milestones 
have been met (See section L.): (1) A model system has been 
built; (2) The system has been tested with a small group of 
users; (3) Plans have been made to implement an operational 
version of the system,; (4) Preliminary planning to market 
the system has begin and (5) Plans hav€^ been made to test 
the IIDA system with various potential user groups. 

The measures which can be used to evaluate the success 
of our proj.ect team m teaching these goals fall into several 
categories: (1) Archival measures — to what extent have we 
met our projected goals. This^can be done through simple 
paper and pencil records; (2) Unobtrusive measures — these 
measures will attempt to test for the initial effectiveness 
of our system with various liser groups; (3) Technical measures- 
those designed to measure actual system performance -from a 
technical perspective; and (4) Impact measures — the effect 
of the system on the users who have *been exposed to it up to 
this point. Each of these formative evaluative measures is 
designed to help us in the further development of the system. 
In addition, they provide indicators to the National Science 
Foundation .for the progress made to this date. 

IIDA will need the benefit of formative evaluation 
during the early years of development and initial use: / 
feedback on how ^ell it is able to teach, on user -acceptance, 
on the techniques developed to promote its extended use, and 
on user reaction independent of its actual success in affecting 
^user behavior. Once the system has been developed and , . 
initial use begun, limited summative evaluative evaluation 
can be conducted on a number of bases which vk^ll be gradually 
expanded to include what will, we hope> be the ^ long range 
evaluation of the impact of IIDA. Long term, summative 
evaluation will be based upon: (1) The ef f-ectiveness of 
teaching: Have user^ learned to search acceptably well? (2) 
Use of the system: Is IIDA actually being used? and (3) The 
user population: Have we, in fact, attracted new users to 
searching science bibliographic information rather than 
converted ot'hers to a new mode of searching? This section 
of the report discusses a number of considerations in the 
overall evaluation planning for IIDA and the specific plans 
for both formative evaluation prior to initial use and 
^summative evaluation plans for the initial user group'. 
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4.4.2 General Considerations 

In general, we feel that It Is helpful to think of 
evaluation, as being tied to distinct levels 'of generalizabxUty; 
I.e. the results can be applied (with confidence) to some 
level of generalizability. At this time, for example, prior 
to full-scale testing with any iiser group, we cannot generalize' 
our results on teaching effectiveness beyond the limited 
number of searches made on a piiot basis .fey students at 
Drexel University. However, in terms of the' technical 
development of the system we ^can make statements at a higher 
level of generalizability. 

Thus, in terms oJE our own - thinking , we distinquiSh be- 
tween statements concerning the technical capabilities of ' 
llhe system, and statements concerning the impact on students 
2nd the ef f ectiveness'of IIDA as a teaching and consulting 
ethods. ' - 

In terms of the overall development of this system, it 
may be useful to thinking of the interaction of system ' 
development and evaluation in terms of a ladder of generalizability 
(see Figure 4.1) As indicated in Section 4.2, we have 
shifted our initial emphasis on user groups from industry to 
the university. Indeed, in testing for the effectiveness of 
IIDA, we will use university sjtudents at Drexel from the 
engineering school as our first user group.' We then hope to 
move to students from the five engineering schools who are 
part of the Experimental Partnership for the Reorientation 
of Teaching. From there, it will be possible- to move^ to' 
testing students from other disciplines, Then^ we can begin 
large-scale testing the IID. i syoten on groups of industry 
users — first '^ne industry and then others. Preliminary 
testing with industrial groups will go' on in parallel with 
academic . testing . 

Clearly, as^ we move from step one on the ladder to step 
five, we can have greater and ''greater confidence concerning 
the statements we are able to make with respect to the 
effectiveness of IIDA as a teaching method and the impact of 
ipA on the productivity and efficiency of individual users. 
Similarly, as we move further up the ladder we can move to , . 

more complex and sophisticated methods of evaluation. 

In terms of the technical development of the system, we 
can also think of levels of generalizability (see Figure 
4.2). At the beginning of the project, IIDA started as a 
model (Section 5). We then tested the model, and are now 
ready to build a full system* With the full system in place, 
we must then begin to test the system with our designated 
"potential users". Finally, we can begin to perfect the system 
and add degrees of complexity into it. 

In terms of the evaluation of this project, it, should 
be clear t^t the IIDA system must be developed to the implementation 
phase (the building of a full system) before full scale evaluation 
(pf a formative or summative type) can be launched. 
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4.4.3 Evaluation Variables y^ 

As the discussion* to this point Indicates, most of our - 
evaluation *ef forts to this point have been in the form of 
planning for the future. There has been some evaluation of 
use of the fiDA model and this Is discussed In Section 5. 

. . It should be^ clear that our dependent variable In this ' 
case can be o^eratlonall^ed through the use of computer . 
records. Some' examples of\ the type of information which can 

'be derived from the computer records kept by the JIDA system 
include: ; . . 

I. Connect time 

II. Search. time 

ill. Connect time/search' time 
iv. Kinds of user errors 
V. Number of user errors 
vi. Kinds of commands used 
vil. Number of commands used 
vlll. The way commands are used 
Ix. User familiarity with commands 
X. Sequences or patterns of errors and commands 
xi. Focusing of search (e,g., search directed commands/undirected 
.commands) 

xll. Repetitions of instructions 

xlll. Repetitions of command- descriptions 

xiv . Frequency of' HELP ' 

XV. Kinds of HELP used ^ 
xvl. Patterns of HELP used ^ ' 

xvii. Changes in kinds of HELP over time 

xvlli. Prompts or indications by IIDA of availability of HELP 
xlxr Number of suggestions' by IIDA 
XX. Kinds of suggestions by IIDA 
xxi. Patterns of suggestions by IIDA 

xxll. Changes in kind s- of suggestions over time • 
xxlil. Growth of completing of search strategy 
xxlv. Costs of searching 

4.4.4 Future Evaluation Plans 

In general, both the formative and summative evaluations 
of IIDA at different stages of the development of the project 
can and should be conducted using several different types 
of techniques for measuring the variables of interest. ^ The 
.kinds of measures which" seem applicable are (1) observation , 
(2) self-report , (3) and records of student and system performance 
over time . In a number of cases some of these kinds of 
measures can be collected unobtrusively so that the user is 
unaware of the fact that measurements are being made or so 
that while the user may know that measurements are being 
made he will not necessarily know what the measures are 
intended to measure. 

!• The dependent variable . As was the case in testing 
the IIDA model, we plan to continue to test for user satisf ication 
an skill. This will be done through the sur^rey instrment 
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and through some interviews conducted by project staff with 
users concerning their judgments About the problems with the 
system. In addition, we will have project staff members, 
monitor the inti^ractions between users and IIDA. The staff 
will determine what aspects of the program users are having " 
difficulty. However, as the user group become^s larger we^ 
will have to rely more upon the computer system itself to 
monitor interactions with the system. 

However, with the full system in. place, we plan to add 
an important dependent variable: impact upon the users biblio - 
-grapbic searching ^habits . Do users continue ' to use IIDA? 
Do they use other computer based bibliographic searching 
techniqul^ How do they compare the relative efficiency of 
IIDA versus other methods? • Is their productivity arid efficiency 
increased through (or influenced by) the use of IIDA? 

This very Important dependent variable will be operationalized 
through a combination of survey research techniques, observation , 
and labor tatory'' experiments , The^ survey research techniques 
combined with interviewing will be used to ascertain user's 
attitudes with respect to the efficiency and ^desirability of 
JIDA versus other bibliographic searching techniques (both "\ 
manual and othei; computer bade^ systems. For a small sample 
(N=50) of users, we can also use this technique to ascertain 
information about their productivity. We would propose to 
interview this small sample several months after they have 
adopted IIDA into their regular research habits. These 
users' would- then be questioned cmceming: (a) the number of 
hours they now spend on bibliographic searching; (b) the 
number of hours they used to spend on this activity; (c) the f 
types^ of activities they were able to "tfike-on" as a by- " ^ 
product as a reallocation of time (it, is .also possible they 
wuld take on fewer activities); (d) in the case of researchers, 
the kinds of substantive acti>rities ' they were able to perform 
(e,g», new articles, experiments, reports, etc) 

In the case of repeated uses of the IIDA system, we 
would rely upon observation of users* behavior— rthrough the 
computer system and through monitoring by our project staff. 
This could give us. an indicator of changes in user skill 
over time. We have discussed the specific measures for this 
in an earlier part of this section. 

It would also be quite useful to conduct 'a laboratory 
experiment with the consortium of schools who havef agreed to 
be potential users of this system. Similar experiments 
could be conducted in industry, ^ 

The experiments would be designed to give the same 
search assignment to a group of approximately 75 potential 
users,' 25 would conduct the search through IIDA; 25 through 
some on-line bibliographic searching system; and 25 through: 
a manual search. These three groups could be compared along 
many of the variables described above: length of time for 
conducting search, number of titles found; number of relevant 
titles found, satisfaction, skill in searcliing, etc, 

/ 
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' ^ We would then propose to follow-up .with each- group and 
test for the impaqt of this experience upon their overall- 
searching procedures. ' ^ ' * ^ . ' ' 

^ Such a laboratory experiment would be quite costly in 
terms of resources and staff . We are, therefore, simply 
. outlining it at this point in time as one possibility for 
future evaluation. 

4.4.5 ' Factors which may influence user's agtiitude's 

In formative and ^summative evaluations conducted through- 
out the various stages of IIDA development and use 'extension 
we must concern ourselves with the effect of IIDA oh the 
user. It is also necessary to recognize that a number of 
characteristics of the user, the system, and the interaction 
between the two are likely-to make a difference,/ from one 
user to the next, -in how 'the system influences the user's 
attitudes about> and his ability to learn from and effectively 
u3e IIPA. So- far we have been able to identify a number of 
variables which may influencje user responses to IIDA. While 
not all of the^e independent variables will b^ of Importance 
in all stages of formative and summative evaluation of IIDA 
it is desirable to h^ve -them listed in advance for use as a 
guide in planning both kinds of evaluations; 

1. Physical characteristics gf IIDA . 

'i. Characteristics of the terminal 

ii. Ease o^ access ' / 

iii. OReliability 

2. Operating. characteristics of IIDA . 

i. Content of tutorials and ex^cises 

ii. Sequence of exposure to tutorials and exercises 
lii. Form of messages 

iv. Nature of feedback to users 

V. Management and control of^Wessages^ 

vi. Time and speed of message handling 

vii. 'State of computer and phone link ^ „ <h 

viii. Access procedures 

ix. Costs 

3. Interaction between IIDA and user . 

i. Specificity and clarity of operating principles 

ii. Frequency of searching 

iii. Amount of pre-planning for search 

iv. Training procedure^ fpr IIDA use 

V. Ease of access to relevant literature via IIDA 

vi. Ease of access to, alternative search systems 

vii. Quality of results with IIDA 

4. User . / . 

i. Physical characteristics (e.g., lack of typing skill, 
physical impairments, etc.) 

ii. Familiarity with terminal, computers, interactive 
searching, etc. 
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iii. Attitudes towards learning and working with computer 
assistance , 
* iv. Personal incentives 

V. Personality characteristics 

vi. Work and life style 

vii. Perception of need or des;irability of working with 
JL;|:terature | ^ ' 

viiij. Strength of desire to be "in touch" with litjerature 
ix. (Attitudes towjard other users (e.g.^ colleagues' or peer 

group) / 

X. Perception of self with respect to other users 

xi. Attitudes toward task \ . 

xii. Personal search patterns — i.e., preconceived or 
acquired notions of how searching "should" be done. 

5. The nature of the task or search . 

i. Divisible v£, unitary 

ii: Maximizing vs. optimizing (most information vs. 
best information) 

iii. Specificity- of goal 
X • iv. Number of alternate methods of goal attainment 
V. Time constraints 
vi. Need for closure or completeness 

6. Social or group influences . 

i. Colleague or peer group — as implicitly defined by the 
user— Attitudes toward IIDA ' * ' _ ' 

ii. Colleaguia or peer attitudes towards searching and 
the need for it , ^ 

iii. General organizational structure of the group ' 

iv. Indiv.idual*s role within the group 
V. Organizational climate 

vi. Group size 

vii. Group definition of Importance ^f the task 

viii. Reasons for the existence of the groups 

ix. Financial limitations imposed by the group, 
ultimately, a s\immative evaluation should be conducted 

to test for the Impact o£ IIDA among diverse groups of 
users-- with different levels of system comple3?ity (see Figures 
4.1 and 4.2) For. the laoment, we have concentrated uppn 
formative evaluation for purposes of system development. 



4.5 Applications of IIDA 

In developing IIDA it has been our belief that the concept is 
general in nature and is applicable to a wide variety of problems other 
than the teaching bibliographic searching. However, since IIDA is not 
the only project which Impinges upon the possible uses of IIDA or IIDA- 
like systems discussed below, it is reasonable to believe that in time 
the various techniques' will be fused and the resulting syst;ems will not 
be exactly the same as any of the present day ones. In this sense we 
view IIDA as a precursor of a family of systems that will be implemented 
to help people use computers and devices which are attached to computers. 
They will save countless hours of dull and often ineffective oral instruction 
or the publishing and reading of ineffective texts. 
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4.5.1 Training of intexmedlarles 

Although IIDA is being developed initially with a single primary 
user group in mind — the end-users of scientific and technological 
information — we believe it will also be of use in initial and refresher 
training of prof ession^al intermedfiaries^. These intermediaries receive 
varying amounts of training and varjring degrees of on-the-job ej^perience 
in bibliographic searching. 'The small likelihood of the intermediaries' 
being thoroughly familiar with any given discipline is one of the rea- 
sons why some end-users of the services they can provide do not like to 
use intermediaries or libraries. If this lack of f aipiliarity with the 
discipline of the end-uger is coupled with inefficiency' in searching- 
owing to inadequacies in training or to lack of experience with searching 
the problem is hade more severe. .-However, the fact that the inter- 
mediaries are information professionals and have some, training should 
make them receptive to IIDA-like training and to the use of the IIDA 
diagnostics while performing routine searches for clients. -It is our ^ 
intent to continue the project as originally designed and, in addition, 
to bring in occasional professional intermediaries for testing, .this 
will allow us ^o .guage their receptivity to pos&ible eventual use of 
IIDA. It will also enlble us to begin building up performance data oa 
experienced searchers relative to non-experienc§d searchers as part of 
our attempts to 'continualJ.y refine and improve the performance of IIDA 
in helping users learn not only the mechanics of searching, but also how . 
to search more effectively. 

,4.5.2 Training of IIDA users in the use and appreciation of their 
professional literature v - ~ - 

In general, science students receive little trailing in the structure 
of the literature of their fields and its use. Working scientists have 
probably picked up some of this on a hit-and-miss basis but do not, in 
general, have the time to fully investigate lit,erature structure and use 
through brute force methods! I^ile IIDA, as the design is presently 
projected, will provide training in one aspect of the use of professional 
literature, there is no coverage of how a literature is organized, how 
publication operates, or how a research person keeps up with a field 
Cnot, by any means entirely through searching or use of a library). 

This leads us to believe that one reasonable extension of IIDA 
would be to deVelop course material with practical exercises on the 
structure of the' literature or the simulation of the publication and 
diffusion of knowledge might be developed. At the undergraduate and 
graduate student 'level these could be added to the curriculum of various 
sciences as assignments that can be given for independent study. For 
the working scientist who wishes to learn more about these topics, the 
availability of these materials though IIDA would provide the opportunity, 
if desired, to learn or work through the material on an ad lib • basis at 
a relatively low cost in time and effort expended. 

4.5.3 Coping with language differences 

Differences in the - languages used for cataloging and indexing have 
plagued all library automation efforts^ Attempts have been made to 
'translate' between classification schedules or thesauri, but not with 
any notable success. Currently attempts are being made to develop a 
.universal search language. There seems to be no reason why this should 
not be successful since the functions jof search services vary relatively 



^ little even though the externals appear tot vary widely. Hiven that 'a 
universal language can he developed, it could be implemented either by 
^ ^,the individual search services, in lieu' of or in, addition to, their own 
languages, or the universal language could be translated byj an inter- 
v^mediary computer, such as that of IIDA. At the same time of course, the . 
other IIDA services of teaching and assisting in -bibliographic searching 
could be provided. - 

. Regardless of 1iow a universal language is brought into being, there 
■"^ is no conflict between such a language and the use of IIDA. IIDA can be 
^ used to 'teach and assist in the use of" whatever language students and 
search services may wish to emp5.oy. Farther, if the approach to a 
universal languag^ is made by using a translator from it to a vendor 
language, tl\en it can be expected that*' there will be areas of ambi^Xilty 
in the translation. If there are, then the IIDA interactive techniques 
will be exactly ^what are heeded to assist in the resolution of the 
.ambiguity. , >^ « ^ * 

^•5.4 Teaching and assistance in other subj ects ^ 

An IIDA-like, system could be tlseful in teaching and assisting in 
• the use of any highly structured activity performed in a computer or 
which is controlled by a computer. Examples of such activities are: use 
of statistical packages (sets of computet programs) on a computer,, 
especially by non-^^xperts in the' field, who are using the' package 
programs for output," not for learning how" they functioif; performing 
control functions through a computer, such »as laboratory experiments and 
analyses whereby the intermediary computer assists in deciding what to 
do; and in the use of large scale, simulators . In many fields of research, 
statistical ^nd simulation programs ^are used, much as -were slide rules 
formerly. Similarly, many laboratory procedures can be either simulated 
or controlled in actual execution through a control computer. In all'of 
these cases the user wants to process data and i^se the results as 
quickly as possible. Individualized instruction plus assistance on-line 
is an ideal approach. Using computer assisted teaching and mediating 
programs such as IIDA can save time and materials as well as reduce 
•errors and turn around time while increasing capacity and flexibility 
for the individual Iiser. 1 



'4.6 Dissemination of Information about IIDA 

Some of our plans for disseminating information about IIDA have 
already been discussed under other headings in .Section 4. These in- , 
yolve: (11 the development of user-oriented workshops and case history 
material; C2) the gradual expansion of the user base through the de- 
velopment of a loyal user group and a record of prov^en performance; and 
C3) the development of an industry-oriented marketing campaign. One 
aspect of this third route for dissemination of information not dis- 
cussed above is continuing professional education. Drexel University 
has a long history of being industry oriented both through its cooperative 
education program and through the development of a broad spectrum of two 
or three day continuing education 'seminars to introduce a number of 
topics to people working in industry. It is likely that at some stage 
of IIDA development we might wish to make use of this mechanism to 
introduce IIDA and its capabilities to potential users in industry. 
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■ ■ ■ ■ ' 5 THE MODEL SYSTEM 

5.1 \Obje(!rtives of the Model ■ k * ^ 

a\^11, working model of IIDA has been developed with two ob- ' 
jectlves;* (l)'to demonstrate that the techhiques proposed for IIDA are 
feasible Wd (2) to^ aid in the design of the full system by requiring 

'designersXto face and solve .operational, prqbleins which one might tend to 
gloss over if no operating model were recuired. It was not in objective 
of the model system^ to provide an educational service to any users. The 
model is not a complete teaching system and, as a result, it cannot be 
used to measure- the ability of IIDA to teach. 

For the sake of economy of operation and administrative ! simplicity , 
a data base processor written by a .member of thp Drexel staff -was used'. ' 
This program^ called IRSYS, emulates, in many ways, the TOALCXp system of ' 
Lockheed. Anyone familiar with DIALOG can^ learn to use iRSYS in a very, 

- few minutes. The file, to be used was a set of records takeiTtfrom Air. 
Pollution Index, prepared by the Franklin Institute Research; Laborator- 
ies for the Environmental Protection Agency. Selection of xhis partic- 
ular file j.7as also on highly pragmatic grounds: ease of*'^ availability , 
knowledge of^ the file by an IIDA programmer who had .to 'convert record 
structure to conform with IRSYS requirements, and the assumption that 

.almost any, test subject or user of the IIDA model system would be* able, . 
to formulate a meaningful question to the file without special knot iedge 
or background. 

A PDP-lO computer was used for all thiee processors, with the 
interaction between them being simulated. Thus, CP, IP and DBP were 
each designed ^^to operate as if the other processors were in.a separate 
computer. ^This slowed response time somewhat, but timing was adequate 
for demonstration purposes. . ^ ^ 



5.2 The Model Curriculum 

J Two typical programs and. an exer-^ise were implemented. 
Jgbrials were data base structure and .earch mechanics.. 



The tu- 
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5.2.1 Tutorial programs 

This portion of the model ^serves to illustrate how tutorial mater- 
ial in general will be designed in the full implementation. The subject 
matter in 'this case was restricted to explaining the commands and data, 
base used with the model DBP. Some use was made of t;he IIDA capability 
for the IP to compose commands to the DBP based. upon what the student 
has input or on what the course author needs. For example, if the 
course author*, wanted to display an example of a thesaurus^ display, 
rather than laboriously preprogram the entire display, he could cause 
the IP to issue an EXPAND command to the DBP and the display was gen- 

^ ' . - 135 ^ • ^ 
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^ Frame 1 



Frame 2 



Frame 3 



Frame l\ 



-V- 



i r> Frame 5 



Frame 6 



Frame 7 



FrdiVie 8 



Frame 9 



Fr-ame 10 



Frame 11 



Enter free text ^statement of search topic, to be stored 
for later analysis, v 



Record, not in computer, descriptors student plans to use. 



Sta£2 1. Winter only the commands SSLECT, EXPAND or 
RELATE. Descriptors chosen by the searcher. 



Verify that at least one of each type of command called . 
for in Stage 1 has been used. If so, pass control to 
Stage 2, If not, ask student to use the missing commands. 



Stage 2. Enter only CCMBIXE commands, using sets created 
during Stage 1. 



Verify that at least one new set has been created with a 
COMBINE command. 



Display set history and choose from among these^ choices: 
restart, back to stage 1, back to 5;t^ ;e 2, go on to 
Stage 3. 



Sjtage- 3, ' ' Enter only PRINT commands. A relevance judgment 
will be called for for every record printed. 



Offer student option to review more sets or to proceed. 



Student assesses nis progress, elects to go back to an 
earlier point or go ahead. 



Student assesses entire search 



End 
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crated and transmitted from the DBF to IP to CP to student terminal. 
Normally, the kinds of student responses were so well restricted by the 
teaching material that the diagnostic programs are not needed. 

5.2.2 The exercise 

The goal of the exercise was to give the student the ex:>erience of 
conducting a search of his choice with tutorial guidance and assistance. 
For this reason definition cf the search topic, descriptors used and 
depth of the search are under the control of the searcher. The system 
maintains control over the order of search stages and constirains the 
student to sample each type of search command at appropriate times. The 
vital HELP function is available on request, and analysis of\ungram- 
matical or inapprop late commands is automatically performed.! 

The order of search commands follows the model of a search phases 
discussed in Section 1. These are:^ (I) exploration of descriptors 
available :md choice of which to use, (II) combination of descriptors 
into more complex concepts, and (III) browsing in the resulting groups 
of Items and print irg of items of choice. 

The student is allowed to leave Phase' I wh,^.r* e co.^siders himself 
ready, subject to the following conditions: (a) may only pass fr.j 
Phase X into L.^se II and (b) he has used, at lea^c in a cursory manner, 
6ach of the search commands available in Phase I. Since the model 
recognizes the iterative nature of interactive searching, when the 
student feels ready to leave Phase II, he is offered several options for 
which phase to enter next. After insuring that the searcher has sampled 
the commands of Phase II, the system offers options ranging- from start- 
ing all over again (prior to Phase I) to continuing ^n without recycling 
(Phase III). In Phase III, the system requests revelence judgments 'for 
each of the records i^rinted. If the searcher feels unsatisfied with 
Phase III, another opportun'^.ty to return to an earlier phase is pre- 
sented. This option, is presented in recognition of the "..ncreased in-* 
formation concerning. the search available to the student c\ completion 
of reviewing some of the results of his search. It is also possible to 
decide the search, is finished on completion of his first, excursion into 
Phase III. 

Using the guided exercise, the student is able to complete a search 
of his choosing to his own satisfaction. Help is available on request. 
The orderly progression of the search' from one phase to another is 
insured without limiting the .uherent flexibility of interactive search- 
ing. By requiring the student to try each command type. in th^. context 
of a real search, we give him a better feel for their value than we can 
by out of context descriptions of function. A simp^lified Jlow chart of 
the guided exercise is shown in Figure 5.1. , Figure 5.2 shows, in gen- 
eral, how diagnostic subroutines are^ used to monitor individual student 
commands . 

An input, presumed to be a command, is received (a) from the 
student by the CP* It passes it to IP where the first check made (b) is 
to see whether it is a 'slash' command, i.e., one ordering a change in 
the sequence of instruction — indicating that the student needs help or 
wishes to stop the course, for 'example. If it is not such a command, 
the input is sent (c) to the DBP. 

The DBP response^ is received (d) by the IP (via CP) and is parse^l. 
If the response indicated (e) that the commaiid was invalid, which is 
denoted by a message beginning with ERROR, thfe ommand is parsed (f>. 
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Figure 5*2^ Schematic of model diagnostic logic, 
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B^s.ed upon the analysis of the command ^ the fexact nature of the error Is 
determined and the' student is asked (g) to reenter. it. This conversa- 
tion may repeat several times until he finally sends a valid command or 
is referred to the human proctor and is terminated from /the teaching 
system. ' - - / 

If the comioand parser program is able, finally, to get a valid 
command, it goes to the DBP and the latter' s response is parsed, as 
above (h).^ Presumably, since the command this time was verified before 
it- was sent, no error message 'will result. /' 

When a valid command has been executed, the results of it are 
entered (i) into the student data base containing the record of his 
performance. Then, the IP determines (j) ,what it wants to do next — 
where to branch to provide tutorial material or to elicit another stu- 
dent' input.. This, branchinj^ decision may be complex. It may use any of 
the information in the student data base .to assist in making this de- 
cision.. ' ' 

The guided exercise is composed of 11 frames. The first two* frames 
require the student tc^ define the search Vo^ic and put in writing the 
descriptors considerefd most likely to be useful in the search. In a 
non-teaching environment, these steps are usually accomplished before 
logging on to the search system. IIDA includes these steps to ensure, 
some ^prior preparation before beginning a search. 

Frame 3 presents the mechanism for Phase I, browsing the thesaurus, 
confirming the presence of desired descriptors, and selecting those 
descriptors which define the scope of the search. The commands to be 
used are SELECT, EXPAND, and RELATE. ' (RELATE is the equivalent of a 
Lockheed EXPAND following an earlier EXPAND calling for display of 
semantically related terms.) 

Certain frames are present in the guided exercise for checking 
purposes. <rhe student is unaware of these frames unless he fails to use 
one of the allowed commands in the proper frame. Frame 4 checks to see 
that the searcher has used all of the commands available in Phase I 
before allowing him to proceed to Phase II. 

Frame 5 enables the operations of Phase II, the combining of des-t 
criptor sets formed in Phase I to create more complex search stafe^raeats. 
The only search command to be used in this frame is COMBINE. Frame 6 is 
a checking frame to make sure the COMBINE command has been used at least 
once before allowing the searcher to proceed. " ^ 

^ A branching node is found in Frame 7. The searcher is provided 
several options consonant with the nature of interactive searching. The 
search may be restarted or a new search initiated (return to Frame 1) ; 
more descriptors may be selected or more browsing through the thesaurus 
allowed (return to Frame 3) ; additional COMBIIfe couTmands may be entered 
(return to Frame 5); or the search may proceed into Phase III (Frame 8). 

The only search command available in Phase III , (Frame. 8) is the 
PRINT command. The searcher requests printing of one of his sets. The 
system requests a relevance judgment on e£xv-h " record before the, next 
record of the set is printed. Frame 9 allows the searcher to, enter' a 
PRINT command for another set, and switches control back to Frai^e 8. 
Frame 8 again requests relevence judgments of the records of the set. 
The searcher indicates when he is satisfied with the number of records 
viewed. 

Another braii^niiig node occurs in I*rame 10. In this frame the 
searcher may return to Phase I (Frame 3), Phase II (Frame 5), Phase III 
(Frame 8), or declare the search complete. Iflieri the sea: ^ is declared 
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complete, Frame 11 requests a relevence judgment on the seardh as a- 
whole. Following entry of the relevence judgment, the guided exercise 
is ended. 

When the searcher requests help at a branching node or at another 
input time, a choice i^f arjaas for help is offered: 

1. More ins t pact ion on what to do now. 

2. Definitions of valid commands. 

3< Reveiw of commands 'and search' stages. 

4. Review of sets created. 

5. Review of ''records already viewed. 

6. Review of " scriptors used. 

Option 1 givfe^ meaningful comments on each frame of the guided 
exercise. Different assistance will be offered depending on what search 
stage the searcher is currently in and will take into account whaf 
progress has been made. Option 2 is tutorial in that valid search 
commands are explained on order. ^ 

The review of commands and search stages provides a listing of all 
the commands isslied-up to the poJ^nt of requesting help. The display is 
in the following format : 



Stage 


Line 


Conunand 


1 : 


1 


EXPAND SMOKES 




2 


SELECT COSTS 




3 


RELATE ENGINES . 




4 


SELECT PARTICULATES 


2 


5- 


COMBINE 2 +"4' 




6 ■ 


COMBINE 2*4 


3 


' 7 

8 \ 


SELECT POLLUTANTS 




SELECT REGULATION . 


4 




PRINT 5/3/'2 




I. \\ 

\ 


PRINT 8/3/5 



A stage is identified as a string of commands all belonging to the 
same search phase. For example, the commands of stags 3 are those 
appropriate to phase I. The stages are numbered consecutively. Using 
this table as a reference, the^system discusses each stage. First, tfie 
stage is identified and a summary of the actions of the sfage is given. 
For example, in stage 2, sets were manipulated with logical operators. 
Searcher input is requesued as a free .text comment on what was being 
accomplished in this stage and searcher satisfaction is queried. If the 
stage seemed unsatisfactory, the searcher is asked his opinion of why. 
After reviewing each stage, the system queries the searcher on his plans 
for continuing and aski to which point in the search he wishes^ to be 
returned. 

In requesting the searcher to identify his goals in each of the 
stages, the searcher is presented with an opportunity to rethink his 
strat-egy and arrive at his own solution. The potential for system 
comments on misuse of certain comm'iands is clear. Examples are trying to 
COMBINE s^ts with no items, or to SELECT descriptors that do not exist. 
However, ^stem directives on which command or descriptor to use next 
are out ol keeping with the philosophy of enabling the searcher to solve 
his own problems. 

The review of sets created 'includes an analysis of sets created by 
the .COMBINE command. Such sets are presented with the original des- 
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criptors, not 

SET - COUNT COMMAND DESCRIPTION 
9 2 , COMBINE 3*8 

/ (1 * 2) * (5 -H 6 -H 7) 
= (COSTS * POLLUTANTS) * (SMOKES + 
ENGINES -H SAE AUSTRALAS)^ ' 
^ The searcher picks out what it is about the sets formed that dis-- 
satisfies him. 

!• Too many items. 

2. Too many irrelevant items 

3. Too few items 

4'. Expected items did not 'show. up. 
The suggestions from the system at this point are typefied by .the re- 
sponse to dissatisfaction with a set with too many items. Suggested 
changes in strategy include the use of RELATE' or EXPAND commands to find 
items more e'xactly on the search topic and the use of the logical AND(*) 
to eliminate extraneous items. 

The review of records viewed first offers a'short analysis of the 
command which generated the set viewed. A detailed definition of a 
COMBINE cominand much like tho : of the review of sets is included. If 
the searcher wishes to view the records again, he will be queried on 
relevence judgments and informed if his evaluation has changed since the 
last viewing. The average relevence for the set is recalculated and 
compared to thfe previous average for the searcher^s information. 

The review of descriptors used displays each of the descriptors 
used and how they were used. 

DESCRIPTOR USED IN SEEN IN ' USED IN 

USED SELECT EXPAND RELATE 

■COSTS X 

PARTICULATES X X 

POLLUTANTS X 

SMOKE X 

ENGINES X 

SAE AUSTRALAS X . X 
Since the system keeps records of what has been seen, meaningful com-- 
ments can be made at thii point regarding missteps such as selecting a 
descriptor which does not exist when the searcher has viewed- t^at* sec- . 
tion of the thesaurus where the descriptor would appear if it existed. 
The need for verification of. descriptors is stressed. 



5.3 Model Software 

The model has been implemented on a PDP-10 computer, wtth all 
processors in the samp physical machine, but communicating among them- 
selves in the manner they would if in separate hardware "structures. The 
programming language has been BASIC which is not ideally ^suited to this 
form of programming, but was a common denominator among project staff 
members and 3 the language in which the model data base processor was 
previously written. 

5.3.1 Instructional processor 

No special system software has been used in the model. Rather, 
courseware has entirely been wi^itten in Extended BASIC. The instruc- 
tional processor structurally consists of a short trunk program sup- 
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ported by an extensive group of diagnostic subroutines. The. trunk s 
function is to cue and cotitrol command input and to call subroutines as 
they are needed. The most prominent subroutines are listed below. 

SLASH ' OTHER * 

SUBROUTINES DIAGNOSTICS 
SLASH DEIECT OR STUDENT FILE UPDATE- 

/no^ COMMAND PARSER 

/Quj^ DBP RESPONSE PARSER 

/jj^LP PR™"^ ROUTINE 

A "slash command" is acceptable whenever^ a search language conmand 
is expected, and at other input ^ times . Valid slash "T^Jj^f ^J"^^' ^ ' 
/DONE, /QUIT. /HALT, and /HELP. The sUsh detector subroutine follows 
each command input. If a valid slash comBand were entered. 
transfers to the appropriate subroutine. A code i-^^-^^J^^ ^^^^^J^^^^^" 
command was entered is sft ..fore control returns to the instructional 

processor^ ^ /pp^ con^d is used by the student to indicate the end of 
any intr a-frame cycle. It is not a subroutine, strictly speaking, since 
its processing only involves the IP branching out of the frame when the 

■/DONE code is recognized. ' , c^i „ „r.™,ia^oH 

2 The /QUIT command clears all pointers and, files accumulated 
on the* student search and terminates the program. Thus t;he student 
begins a completely -fresh search the next time he inters the system. 

3 The /H ELP- command calls in the program's largest diagnostic 
sub-routine. It in turn depends on other subroutines for its °P^^^<;i°"- ' 
Tive options are initially available with /HELP: (D further instruc- 
tions for the current frame; (2) definitions of all valid commands, 3) 

a review of commands and search stages; (4) a review of sets created}- 
and (5) a review of records already viewed. 

In more detail: ^ c ^x. 

■ (1) Paragraphs corresponding to each frame of the trunk are 
stored in /HELP for its first option. These paragraphs describe in 
detail what is. expected of the student in each frame, ^"len this 
help is .requested, the appropriate paragraph is sent tc .he student. 

(2) Paragraphs describing each of the five commands valid in 
the model are stored for the second optiW. These paragraphs 
explain the commands' functions, abbreviations, argument formats, 
and expected data base responses. The student selects one or more 
command(s) whose paragraphs, are then sent to the student. In the 
full system, all search commands will be described. 

- (3) All valid commands given in the search up to this tim^ are 
printed in a table. They have^been analyzed into three stages as 
described by Penniman, and into several transitional stages in 
which commands are mixed. The student is asked to state his ob- 
jectives for the commands in each stage. If he feels that those 
objectives were not met, he is asked to state. why not. Wlien the 
review is completed, a plan for proceeding with the search is 
requested from the student. All prose statements are cap.tured on 
the communications log for later analysis by the human P^'^tor. 

(4) Data about the created sets is printed in a table. This 
includes set number, count of records J-.n set, command ^ext ana- 
lyzed COMBINE arguments, ^and average relevance of ^^^^^^ J^"' j^^f " 
student is asked to indicate those sets which came closest to his 



. search objectives. It is assumed that if any sets were fully 
satisfactory, this /HELP option would not have been selected. The 
student is thus asked why each qf the indicated sets is not satis- 
factory — are there too many records? too few records? too many 
irrelevant records? etc. Advice on revising . the search strategy 

.is then given, *based on this last set of responses. ^ 

(5) Each set vie^^ed up to this point is described in terms of 
the command which generated the set. If that generating command 
were COMBINE it would be broken domi into its basic textual com- 
ponents. For each such set, the student is given a choice to 
review or not to review records which have already been viewed from 
that set,. If a review is chosen, it will be in the same format and 
in the same range as in the original PRINT command. For each 
record thus viewed, a relevance judgement is solicited. If it dif- 
fers from the previous relevance judgement, this is brought to the 
attention of the student. If dissatisfaction with the set is ' . 
indicated, thjB 'reason for this dissatisfaction is elicited and 
advice is given for proceeding. 

Each of the five /HELP options described above are terminated by 
the. student choosing to: (1) return to the courses; (2) sign off tern- 
porarMy; (3) sign of f permanently; and (4) get more iielp by returning 
to th^ top of the /HELP subroutine,- 

h The student file update, pemanentlv stores data oJi the stu- 
J search activities.^ Data is stored ^foj both valid commands and 
erroris. This subroutine is generally called immediately, following a DBP 
respdnse. Data in this file is used in various diagnostic routines, 
both/ for analyzing and for describing the student *s progress. 

/ Data stored about valid coumands include; 'text of command j command 
cod^; command stage code; set number created or viewed; command argument 
text; indication of multiple use of any argument; count of records in 
each set; average relevance of viewed sets; basic textual components of 
COMBINE commands; text of first and last EXPAND terms; count of valid 
commands; and a relevance-format chain for viewed racords. Data stored 
on errors include: attempted command (if identif iajle) ; error code; and 
where the error occurs in the command input sequence. 

6- The command parser completely analyzes student commands. AH 
command components are broken out and stored in temporary arrays. This 
subrouting can be called by parts of the IP to provide a basis for 
thoroughly discussing the command as given by the* student. Errors that 
occur here are identified precisely and are coded for storage in the 
student file^ 

7* The DBP response processor completely analyzes responses from 
the DBP. All response components are broken out and stored in temporary 
arrays. Cryptic error messages from the DBP cause the command parser to 
be called. This parser provides a basis for a thorough discussion of 
the DBP response. 

8. PRINT routine was developed because the system design requires 
the PRINT command not to be copminicated verbatim .to the DBP. This is 
because the student must make a relevance jadgemenr after each record is 
viewed.* Thus the PRINT command given by the student is broken down and 
PRINT commands requesting a single record at a time are issued to the 
DBP. 

The routine begins with instructions that parse the PRINT^ command 
as it is given by the student. Any errors are explicitly pointed out;- ^ 
data about errors are recorded on the student file; and the command is 
entered again. Following the display of each record, individually 



• i3li r 



requested from the DBF, a relevance rating Is solicited. This, Is re- 
corded on the relevance-format chain. All other elements required by 
the student file are duly posted. 

\-32 2' The data base processor 

The data base processor used in the model is a program called IRSYS 
which was cxcaced by a ^Drexel graduate student foi; other ^Jurposes. It 
resembles the DIALOG system in its command structure, but has fewer 
commands. Some of them are more .versatile than 'DIALOG^s for example if> 
a set number 3 has been previously defined, IRSYS permits, the usage 
COMBINE 3* CAI to enable users t6 make use of both descriptor selection 
and set combination in one command. IRSYS also permits string search- 
ing, in the fashion of SDC^s ORBIT. However, for the purposes of the 
IIDA model, only a minimal language is taught to the student. The major 
weakness,,,.of IRSYS for this application is that imposed on it by the PDP-' 
10 operating system for BASIC programs ~ that only one user terminal 
can make iise of any file at one time. Thus, the model is" a one-ua.er-at- 
a-time system. 

♦« 

5 . 3 . 3 The communications processor 

^ 'In the model, the CP performs a minimal role. Because all the 
programs are resident in the same computer, it is present only to make 
the model design conform with the full system design wherever possible. 
As implemented, all communications between the IP and the DBP pass 
through the CP and all ccmmunications with the users pass through this 
processor also. The GP slso logs all messages it handles, which may be 
used for analysis of performance after a student has done a search. 

Because all three processors are resident in the same computer, the 
CP imposes no significant delay in response by the IP to student input. 
In the full system, inter-compute^ communications will be a critical 
timing factor. 



5.4 Preliminary Evaluation 

In addition t'o the design and mechanical testing of a model system 
we have also done some limited pilot testing of IIDA through, use of the 
model to simulate the full scale system. Our feeling was that it was 
necessary, as early as possible, to begin testing out the operation of 
the model system. This was done for both formative and summative eval- 
uation purposes. Actually using the model with naive users offers us 
the possibility^ of detecting conceptual design flaws in the model which 
might be cartied over into the full-scale system if not spotted in 
advance. - In Addition it makes it possible^ to begin to get a feel for 
user reactions to the system. 

The basic procedure used in testing'the system involved recruiting' 
naive vplunteers who were paid for their time. The volunteers were A 
graduate students from the Drexel Graduate School of Librl^y Science and 
8 undergraduates, 5 of whom were from various Drexel engineering pro- 
grams. The remianing 3 undergraduates were enrolled in behavioral 
science programs on the Drexel campus. In all cases the volunteers were 
given an overview of the goals of the project and how the model system 
fits into long range development plans. Below are the instructions 
given to each user prior to actual use of the system. 

The present IIDA system is a model^of some of the component 

parts of a projected larger system and is designed to demonstrate 



the feasibility of actually developing a full-scale system. The 
full system will provide ajmechanism for teaching bibliographic 
interactive information retrieval. It will also serve as an on- 
line' consultant for users. WW completed IIDA will contain tu- 
torials on such topics as search strategy, the structure of a data 
base, the data base command language, etc; It will also contain an 
exercise which requires the learner to go through the stages of an 
actual search while demonstrating ^working knowledge of the data 
base commands. In addition IIDA wmJ" contain an assistance pro- 
gram which, monitors and , provides on-iine consulting for the searcher. 

Having done some preliminary testing of the model we would now 
like to get outside feedback about the^odel. As presently struc- 
tured the system to be demonstrated consists of two parts, a tutor- 
ial on a command language developed for uSe with tljis model, and an 
" exercise.- The data base available for searching vhile using the 
system consists of eighty records and a mic^o-thesaurus from the 
Air Pollution' Index. * After ypu have 'seen orVwprked through IIDA 
please feel free to give us any comments or. advice -you may have on 
the model, or the projected full system, so tiiat we can improve the 
design of the model, and ultimately," the full system. 

In watching or working through a demonstration of the model we- 
would appreciate it if you could pay particular attention to such 
questions as; How effective is' the tutorial in teaching the basic , 
commands of ..the model command language? How effective is the 
^ exercise- in leading^ the student through the stages of a search? 
How effective is the exercise in requiring the user, to demonstrate 
a working knowledge of the command language? 

J r 

After reading these' instructions , the users were asked to work 
through the material container! in the model — the tutorial on the model 
command language and the exercise using the data base of records from 
the Air Pollution Index. This required, in almost all cases, less than 
two hours. Since at the time the users were run there were still mechan- 
ical and system b.ugs,, a mfember of the IIDA project staff served both^as 
observer and as proctor to assist any user. in overcoming system dif- 
ficulties of a mechanical nature. The .observers also kept track of the 
difficulties encountered by the user so that chahges and refinements in 
the model could be jmade. * * , . 

There would seem, to be little point in making, a detailed report on 
the number of mechanical flaws, misspellings, or confusing vordings 
revealed by these observations , Suffice it to say that they existed in 
sufficient number to cause some chagrin on the part of some of the 
observers who had also been involved in the educational programming of 
the model. However, overall, the consensus of the observers was that 
the system worked as expected and seemed to work w^ll. 

This feeling on^ the part of the observers is supported by the 
responses given us by the users on a questionnaire administered at the 
conclusion of the training session. The purpose of the 'questionnaire - * 
was to give us some written feedback about the users* affective re- 
sponse o to.trhe system and some of their Impressions about working with 
IIDA. The questionnaire consisted of two sections. The first asked the 
users for their feedings and comments a^bout IIDA as. a way of learning 
about interactive searching. The second section asked the users for 
their feelings'' and. comments about *IIDA as an assistant or consultant in 
actually doing on-line searching. It should be noted that in the second 
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part of the questionnaire, the context for the comments made by the 
users is^ one in which their only real exposure to IIDA as a consultant 
is through the exercise* At that point IIDA is in a transition between 
the roles of teacher and consultant. -^X 

When asked to give their positive impressions of IIDA as a teacher, 
all of our users indicated that there wa^ something that they liked. In 
general the aspects of IIDA most preferre<3 'had to do with the flexi- 
bility offered by an interactive system — -"-jElexibility in terms of 
guiding their own learning by being able to/ choose search topics, work 
at their own speed, or do reviews as desired. 

When asked to give their negative impressions of' IIDA as a teacher, 
all but one of our users indicated that they disliked something. In 
general, however, what they reported disliking were certain physical or 
mechanical aspects of the system. Two users did, however, feel that the 
system needed more flexibility in terms of being able to explain some- 
thing to the user in more than one way if he did not get it the firsbyt 
time. ^ 

When asked how their perceptions of learning to do interactive 
information retrieval had changed since using IIDA, inexperienced users 
tended to report either no change or that they had not realized before 
what- was involved in learning about interactive searching. Experienced 
users almost unanimously felt that exposure to IIDA had helped them 
develop more confidence and a better feel for what was actually going 
on. 

When asked to give their positive impressions of IIDA as a con- 
sultant in actually doing a search, almost all users responded with 
comments indicating that they liked the flexibility, consistency, and 
scope of search capacity made possible by IIDA. 

When asked to give their negative impressions of IIDA as a con- 
* sultant in searching, the users, in general, reported disliking some 
physical or mechanical aspects of the system (e.g., format of comments 
made by IIDA) . . Other negative impressions either focused on user lack c 
of knowledge about the data base or the data base thesaurus* The ipajor 
negative comments about .aspects of IIDA per se came from the more ex-^ 
perienced users who felt that* there was too much repetition of infor- 
mation. 

When asked how their perceptions of actually doing an interactive 
search had changed since using IIDA, most inexperienced users eitlier 
reported no change or felt that they now had a feel for the commands and 
how they could be used. The more experienced users either felt it might 
not be necessary to >^we IIDA available or felt that IIDA would be of 
great help as a consultant. 

When asked ix tl^ey would like to sue IIDA as a consultant in doing 
a literature search o^ a topic relevant to their own interest, all 12 
users but 1 indicated^ that they would. The one person who said "no^'^ 
felt that IIDA would be unnecessary because everything she needs is 
already easily available. 

The questionnaire used with test subjects follows. 
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Since you are part of the first group to have any extended experience 
with IIDA, we would like you to record your reactions and comments to help 
guide us in the "future development of this system. At this point we are 
particularly interested in any suggestions you may have for improvement of . 
the system and in what you see as being the potential benefits of a system 
such as IIDA^ ' « ' . 

When fully developed IIDA will in effect be able to serve two roles. 
The first role is that of teacher of the skills .required to do interactive^ 
information retrieval. The ^^ond role is that of an on-line consultant 
to, assist the searcher if problems are encountered. This questionnaire is 
divided into two major sections. In fche first section you yill find a series 
of questions asking you for your impressions of IIDA as a teacher. In the second 
you will find questions asking you for your impressions of IIDA as a consultant. 

On the following page you will find several scales designed to assess 
your feelings and attitudes toward IIDA as a tool for learning how to do 
interactive information retrieval. There are 18 scales altogether. Even 
if you find some of them strange or inappropriate it is important that you 
complete them all^ 

Please.-, consider : '0 
IIDA as a way of learning interactive information retrieval 

You have conducted a search of a mini data base using IIDA. Please 
don't focus on the product of the search itself, but consider "the system 
you used to learn about interactive searching* 

Work rapidly through the scales without pausing for more than just a 
few seconds on each one and without returning to one. you have already completed.** 

V 

Please. place a check at the point on each scale which you feel to be most 
O copriate. j ^ 
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IIDA as a way of learning interactive information retrieval 



boring 
complex 
free 
good 
important 
tneaningless 
passive 
pleasurable 
sensitive 
stabl 
strong 
tenacious 
\ ugly 
unsuccessful 
technical 
relaxed 
informal 
valuable 



: interesting 

: simple 

: constrained 

; bad 

; unimportant 
: nieaningful 
; active 
: painful 
_ : insensitive 

: changeable 

: weak 

: yieldinj5 

; beautiful 
, : successful 
non- technical 
; tense 
: formal 
: worthless 
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Next we would like to have you answer a series of ques-fcions about 
HDA as a teacher or a system for learning how to do interactive information 
retrieval. We would appreciate it if you cotild help us by answering them 
as fully as you can. 

1, How good are your UjT^ing skills? (Please describe.) 

Z. How familiar are you with computers and their uses? (Please describe^) 



3.. Bow much experience have you had in working with computer^ using: remote 
terminals? (Please describe.) 



4. .How much experience have you had with computer based interactive 
information retrieval? (Please describe.) 



5. Have you had any previous experience with other methods of teaching 
computer based interactive information retrieval? (If ••yes'* continue 
with 6, if «no" go to 8.) 

6. Which methods? (Please describe.) 



7. Given the way you first learned to search, what do you think about this 
method? 
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?leas<. describe your impressions of learning to do computer based 
interactive information retrieval urging IIDA as a teacher or system 
for learning. 

Positive Impressions: 



Negative Iwpres&ionsj 



How has your perception of learning to do computer based interactive 
information retrieval changed since using IIDA? (Please describe.) 
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Kow that you have given us your reactions to HDA as a teacher or 
a system for learning how to do computer based interactive infomation 
retrieval we would like to have yc : give us your impressions of IIDA as 
a consultant in actually doing interactive information retrieval. 



On the following pkge you will find several scales designed to assess 
your feelings and attitudes toward HDA as a- consultant to assist in doinp 
interactive information retrieval. There are IB scales altogether. Even 
if you find some of them strange cr inappropriate it is important that you 
compete them all. 

Please consider: 

IIDA as a consultant to assxdt in interactive information retrieval 
You have conducted a search of a mini data base using HDA. Please 
don't focus on the product uf the search itself, but consider the system 
as a consultant. 

Work rapidly through the scale? without pausing for more than just a 
few seconds on each one and without returning to one you have already comp'' ted. 
Pie- 3 place a check at the point on each scule which you feel to be most 
appropriate. 
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IIDA-as a constiltant to assist in interactive information retrieval 



borings 
complex 
free 
good 
important 
meaningless !^ 
passive :^ 
pleasurable 
sensitive :^ 
stable 
strong 
tonacious 
ugly :^ 
Tinsuccessful. :^ 
technical i_ 
relaxed 
informal i_ 
valuable : 
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J : interesting 

% simple 

t constrained 

: t bad 

unimportant 

_J J mer.,ningful 

X : active 

^: I painful 

J t insensitive 

t t changeable 

J t woak 

J : yielding 

J : beautiful 

J I successful 

J In non-techni cal 

J t tense 

: : formal 

: I worthless 
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Next we woiild like to have you answer a series of questions about 
IIDA as a consultant in doing interactive information retrieval. We would 
appreciate it if you could help us by answering them as fully as you can. 

1. Please describe your overall impressions of doing computer based 
interactive information retrieval with IIDA serving as an assistant 
or consultant. 

Positive Impressions! 



Negative Impressions t 



2# How has your perception of doing computer based interactive information 
retrieval changed since using HDA? (Please describe.) 



1.5:3 
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3. If IIDA were available for you to use as a consultant in doing a 
literature search on a topic of relevance to you, do you think you 
would make us^e of it? Why? ' 



mNK YOU FOR YOUR HELP 

(If you have any further thoughts about UDA, either as teacher or as consultant, 
please use the space below. We are particularly interested in any i:ug£restions 
you may have for improvement of the system and in what you see as being the 
potential benefits of a system such as IIDA.) 



4? 



ERIC 



APPENDIX A: TUTORIAL PROGRAMS 
1. STRUCTURE OF COMPENDEX 

1.1 Objectives of the Course 

The purpose of this course is to teach students: (1) about th^ 
material the comprises the Compendex file, (2) the structure of the file 
in the sense of the information elements that make up a record, and (3) 
the use of bibliographic tools such as subject headings. No prior 
background in the use of Compendex or the Engineering Index is assumed. 

1.2 General Characteristics of Compendex 

This section will identify the publisher of Compendex, the materials 
covered by it, size of the file, date span of contents, and frequency of 
updating. 

1.3 Organization of the File 

This section will cover the specific fields of information present 
in a Compendex recorc} and their interrelationships. It will describe 
the formats available for record printing or. display and will show the 
student one or more sample recrods, annotated as to element structure. 

1.4* Abstracting in 'Compendex 

This section will describe how abstracting is done and by whom. It 
will cover the standard content of an abstract, what material is reviewed 
for abstracting, and who the abstractors are. 

1.5 Subject Description 

Here will be described the subject division^ ard subject headings 
used in Comp^dex. Mention will be made, in anticipation of later 
course material, of the possibility of searching titles and abstracts 
for occurence and co-occurences of words. 

1.6 Timing 

(\ 

This course should be completed in 20 - 30 minutes by the average 
student. 
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2. SEARCH MECHANICS 

2*1 Objectives aiid Organization of the Course 

This .course introduces the student to the tools used in conducting 
a search. Some mention is made of the terminal, but concentration is on 
software tools ~ the search or command language. The course will be 
taught in two segments. The first introduces a minimal subset of the 
command language, a small number of commands which can be easily learned 
but which can enable a student to begin doing useful searches almost 
immediately. In some cases, the many variations actually possible in 
the use of a command will not be taught, and only a limited range of 
uses described. In the second segment, the full command set will be 
presented* 

The intent is that a typical student would take only the first seg- 
ment initially , then go on ta the exercises and possibly even an assisted 
search. After he begins to feel comfortable with the basic commands, he 
can return to the second segment of this course and the corresponding 
exercise. For some users, it will never be necessary to take the full 
language segment or exercise. 

2.1 Access to the System 

Ir the first section of the course, the mechanics of accessing 
DIALOG are described: use of the telephone, dialling into the network, 
the DIALOG password, the use of the terminal, mechanical differences 
between DIALOG AND IIDA. 

2.2 The Minimal Subset Segment 

Commands covered in this segment are: BEGIN, EXPAND, SELECx, 
COMBINE, PRINT and LOGOFF. They will be taught with the following 
restrictions: 

1. BEGIN. 'Always used in the form BEGINS which calls for" the 
Compendex file and reinitializes set numbers. 

2. EXPAND . The argument of EXPAND may be a descriptor or a. line 
number (En or Rn) . 

3. SELECT . The argument of a SELECT command in this segment is 
restricted to a descriptor or a single line number (En or Rn) . Truncated 
descriptors, multiple line numbers or full *text searches are not taught'^ 
at this level. 

4. C OMBINE . The argument of COMBINE will be taught as an ex- 
pression of set numbers with boolean operators. Parentheses may be 
used. 

5. PRINT. Printing will be restricted to either of two formats: 
one shov;ing title only and one the complete citation without abstract. 
The abstract is omitted in order to reduce the volume of printing. 

6. LOGOFF. This command has no variations or argument. 



This segment of the course should ^ake a typical student about 15 
minutes to complete. 
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2.3 The Full Language Segment 

In this segment the full range of each command is explained to the 
student. He need not take the entire course in sequence, but will be 
permitted either to do that or to request instruction on a particular 
command only. Thus, this segment of the course can be used in a ^refer- 
ence mode as well as a conventiojial tut;orial mode. 

If presented sequentially, the commands will be described in phase 
order, using the five phases listed in Section 1.2.2 (index search, 
logic formation, record display, procedural and diagnostic). The . 
diagnostic portion will concentrate on diagnostic information available 
'through DIALOG but will briefly introduce the extended set of diag- 
nostics availably through IIDA. 

IIDA instruction will not cover the symbolic form of conmiands 
accepted by DIALOG, except for the ? representing EXPLAIN. 

The second segment of the search mechanics course, if taken In its 
entirety, would probably consume an hour. 
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3. SEARCH STRATEGY 



3.1 Objectives 

By now the student^ has learned something about the composition of 
Compendex and the commands available to him to search it. This course 
will provide instruction on some general principles for conducting a 
search — how to plan and approach tlie overall search. We have pre- 
viously pointed out that there is no accepted theory of search strategy. 
Nonetheless, we hope to teach students from the beginning tha^ they 
should have some plan when they begin a search. ^ 

3.2 Statement of Objective 

\ 

We will ask students to state a search objective. . This is intended 
to be a guide not a limit aiton on their performance. They migh: elect 
to search for a few high grade references, a bibliography of a dozen or 
so citations, or a fairly complete bibliography, that may run to dozens, 
even hundreds of citations ♦ The point is that search behavior will be 
different if the searcher is looking for one or two 'best references 
rather than a hundred or so somewhat relevant ones. The student may at 
any time change his objective, but he should be conscious of the fact 
that he is doing this. 

3.3 Starting Point and Direction 

The student should begin his 'search with a few descriptors in mind. 
He should also decide early in the search whether he will try to find 
one or two articles of known relevanca and use bibliographic descriptors 
found with them or work from descriptors to documents; whether he will 
try to define a small set and then gradually enlarge it to suit his 
needs, or .define a large set and gradually reduce it until it satisfies 
him. 

3.4 Specific Techniques. 

Here, we will discuss a few techniques a user may use to get a 
search started, such as 

Locating a known article and using its descriptors to look for 

more. 

Locating articles in a journal or journal issues of known 
relevance. 

Retrieving articles by a well-known author in a field, or 
originating in a specific laboratory or institution. 

Using date or other limits to reduce the size of exploratory 
sets until an adequate-appearing set description is reached, then 
expanding the date range. | 

There are others. There will be no attempt to be exhaustive, 
rather to implant in the student *s mind the idea that there is or can be 
strategy to his searching. 

15b 
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3.5 Timing 

/ 

This course should* take no more than 15 njinutes. 



15b 



ERIC 



• 150 



4. DIAGNOSTICS . 

4.1 Purpose of IIDA Diagnostic Instruction 

Diagnos^pi^s are the heart of the IIDA teaching technique. Their 
purpose is to help the student identify the source of his owri p'roblems 
by providing analyses from various points of view of his commands, 
results obtained and errors made. The diagnostics cannot interpret what 
the student intended. Except for readily-detected procedural or syn- 
tactic errors, they try to give the student data from which he can 
determine that hp is doing something * wrong* (i.e., unproductive) and 
change his approach. , " , 

Initially, a student might be overwhelmed with diagnoses of a - 
process which he does not yet understand well. Use of the diagnostics 
also requries skill, as does searching, and both forms of skill require 
instruction and practice. Hence, the student must be taught what diag- 
nostics are available and how to use them. 

4.2 Diagnostic Techniques 

« 

4.2.1 Parser . This is the first diagnostic program the student is 
likely to encounter and it is one over which he has no control. When- 
ever, in exercise or assistance mode, a student enters a command that is 
either procedurally restricted at that time or syntactically incorrect, 
the parser will converse with him. In this course, the student will be 
told the circumstances under which this happens, given some examples of 
parser messages, and instructed how to respond. 

4.2.2 /HELP . The /HELP program may be voluntarily involved by a 
student or called by the performance analysis routine. /HELP operates 
on the concept of a tree of menus > through which the student selects a 
general, than ever more specific diagnostic or instructional assistance. 
He will be shown examples of the /HELP menus and the logic of their 
construction will be explained. He has options,! which will be explained, 
for returning to the porgram from whence he camet or to other program 
safter he has invoked /HELP. The use of /HELP as a tutorial program 
will be explained. 

4.2.3 .Performance analysis routine . The course will explain to the 
student the kihds of checks made by the PAR and the circumstances under 
which it invoke^v /HELP. 

4.2.4 Proctor. The role of the proctor will be explained, as will the 
mechanics of how a student may contact him or be contacted by him. 

4.2.5 Restart . In the event of machine failure during a search, the 
student will have the option of continuing his previously begun search 
or starting over, when the computer is eventually restored to service. 
The nature of this option and its implications will be explained to him 
both in this diagnostic course and more briefly, each time a restart 
occurs.. 

luU 



'. . . * 4.3 Timing. ° • 

This course should require about 15-20 minutes 
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APPENDIX B: TRANSCRIPT OF A 'TEST USE 



On the following pages is a transcript of the record of a search 
done by a student during the testing of the IIDA. model. It is pre- 
sented^here to illustrate the kinds of messages a student receives and 
how he might react to them. It is not^^an "ideal" use ~ the transcript 
was selected at random from the complete set of trial uses. 

In a few cases the right hand portion of a line of text^has been 
truncated to -accommodate the wider margins of this report. No serious 
lessening of comprehensibility has resulted. 

The material on the last two pages is a diagnostic sunmiary of the 
search. It is printed out, not for student use, but in 'simulation of 
the kind of material a proctor would want who was reviewing a student's 
performance. 
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f^UNSftV iSOlBED.SftV 
STUDENT NfthE: 

GOOD DRY. YOU HflVK" LOGGED INTO THE GUIDED EXERCI^-'E 

SMftLL DftTft BftSE DH ftIR PDLLUTIOK IS ftVfi-ILftBLE. ^ * <^ . 

•:.DME GUIDfWCE IM SEflRCH-IMG- WILL BE flUTQMftTIC. IF ftT ' 
fi(IY TIME YOU REQUIRE, FURTHER ftSSISTftMCE» fi^PE IH 
/HELP FOLLOWED BY ft CflRRIftSE RETURrf, 

•REMEMBER TO MfiKE ftN EMTRY ONLY AFTER -GETTING ft QUESTIONMftRK 
REMEMBER ftLSD TO END EVERY ENTRY WITH ft CftRRIftGE RETURN <CR>: 
PLEftSE TYPE IN ft SHORT SUMMARY OF YOUR SEfiPCH TOPIC. 



MEftSURING EMISSION SOURCES 

USING- PENCIL ftND PftPER» JOT DOWN ft FEW DESCRIPTORS 

YOU FEEL MftY BE HELPFUL. WHEN YOU ARE THROUGH WRITING, 

HIT THE CARRIft6E RETURN. 



FILE AIRP IS ON-LINE 

SET COUNT DESCRIPTION 



SEARCHING ft DftTft- BftSE CAN BE DIVIDED INTO THREE PHASES 

PHftSE 1 IS THE TIME FOR BROWSING IN THE THESft<JRUs' 
WITH THE EXPAND AfiD RELATE COMMANDS. 

PHftSE i IS ALSO The time for using the select 

COMMAND TO FORM SETS OF ITEMS INDEXED BY THE DESCRIPTORS 
OF YOUR CHOICE. 

PHftSE 1 COMMANDS ARE EXPAND r RELftTEr AND SELECT. 
IT IS USEFUL TO KNOW ALL PHASE 1 COMMANDS. 
YOU MUST TRY ALL PHftSE t COMMANDS BEFORE 
MOVING ON TO PHftSE 2 BY ENTERING /DONE. 



ENTER PHftSE 1 COMMAND OR /DONE. 
?SELECT EMISSION Fft<:TORS 
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ENTER ^PHftSE 1 COMMAND OP ^'DDNE. 1 
?SELECT PDLLUTfiMTS \ 

a a PDLLUTfllSTS ^ 

ENTER PHftSE 1 CDMHfiHD DR /DOME. 
7SELECT PERDSQLS 

3 14 RERDSOLS 

ENTER PH^SE 1 CUMMftND DR /DOME. 
?EXPftND flERDSOLS 



REF 


DESCRIPTION 


CNT 


RET 










12 


19=fiERDSDL ^ftTOniZRTION 


I 


•2 


13 . 


l'?=aERDSDL ELECTRICftL PHEMDMENft 


3 


. 2 


14 


19=flERDSDL FOP-^TION 


3 


3 


15 


19=flERaSDL PHENDMENft 


1 


• 6 


16 


19=ftERDSDL SPECTROMETERS 


1 






-05=flERDSDLS 






17 


19=RERDSDLS 


14 


4 


la 


09=ftIR POLLUTION CONTROL ftSSOC RNNU 


W 17 


0 


19 


• 09=ftIR POLLUTION CONTROL ftSSOC SPEC 


S 1 


0 


- 20 


19=«IR POLLUTION FORECASTING 


1 


5 


21 


19^IR PREHEBT 


1 


1 


22 


16-ftIR QURLITY MEftS 


3 


0 



ENTER PHftSE 1 COMMfiND OP /DONE. 
TRELftTE RERDSOLS 



REF 


DESCRIPTIOrt 


PEL 


CNT 


RET 


17 


1'9=P.£R0SDLS 




14 


4 


322 


19'SETTLIN& PARTICLES 


0 


1 


5 


343 


19=SUSPENDED PftRTICULftTES 


0 


3 


6 


277 


19=PflRTICULftTES 


1 


26 


13 


239 


19»P0LLUTflNTS 


I 


3 


17 



ENTER PHfiSE 1 COMMAND OR /DONE. 

?DDNE- 

USER input: DONE 

ERROR it 14 : CPNNDT RECOGNIZE D AS A VALID COMMAND CODE 

ENTER PHASE 1 CUMMAND DR /DONE. 
?/DONE 

„YOU HAVE NOW ENTERED PHASE 2. 

THIS IS A DISPLAY OF THE HISTORY OF THE SETS YOU HAVE 
CREATED SO FAR. 
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E^"^ COMMfiND DE2CR1PTIDM- 



T 

a 

3 



SELECT EMISSION FftrTORS 
3 SELECT PQLLUTftNTS 
l-* SELECT ftEPOSOLS 



PCLEVf^CE 



iiTTu iH/iJ^^^ ^ "^"^ C^OMBINE COMMfiND IS ftVftlLftBLE TU ll^P 

I J!'^''^^''^ ^^f^" ^^Bl-E SHOWN ftBOVE 

ENTER ft COMBINE COMMAND. "iju.t. 

?C0MBINE 1 2 



0 



ENTliR COMBINE COMMfiND OR /DONE 
?CtlMBINE a 3 
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ENTER COMBINE CDMMftND OR /DONE 

USER INPUT: DONE 

ERROR tt I : INCORRECT COMMftND 

ENTER COMBINE COMMftND OR /DONE 
^ ?/DONE 



2°!:!"^ COMMftND DESCRIPTION- 



1 


6 


SELECT 




3 


SELECT . 


3 


» 14 


SELECT 


4 


0 


COMBINE 


5 


32 


COMBINE 



POLLUTftNTS 

ftEPOSDLS 
1 ^» 



=EMISSION FftOTOPS POLLUTftNTS 
■2 +• 3 , " 

=PDLLUTftNTS ftEROSOLS . 



R^LEVftNCE 
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BRftNCWING TO OTHER PHftSES IS NDW POSSIBLE. 
ENTER DNE OF THESE OPTIONS BY tlUMBER: 

1. SftTISFIED» U»#>T TU PRINT SOME ITEHS TO VIEW BY 
GOING ON TO PHftSE 3- 

2. NEED MORE DEf "RIPTORSr 'WftNT TO" RETURN- TO PHft-SE 1 
AND SELECTf EXPftND? OR PELRTE nOPE TERHS, 

3. NEED "iORE'-LIMITeD SETSr WRHT TO RETURrt TO PHftSE 2 
TO COMBINE MORE SETS, 

4. DISSftTISFIEDr WANT TO START SAME SEARCH OVER BY 
RETURNING TO BEGINNING. 

5. UNHAPPY* WANT TO START NEW SEARCH BY RETURNING 
TO THE BEGINNING. 

6. UNSURE, NEED HELP. -^x 

?3 ' 

ENTER COMBINE COMMAND OR /DONE. 
?CDM^INE 1 -K 3 

6 20 1 + 3 

ENTER COMBINE COMMAND "'OR /DONE. 
■^COMBINE 1-^2. 

7 14 1+2 

ENTER COMBINE COMMAND OR /DONE. 
T/DONE 

YOU HAVE COMPLETcD, AT LEAST TEMPORARILY-' PHASES t 
AND 2. BEFORE PROCEEDING, REVIEW YOUR SEARCH SO FAR. 
THIS IS A DISPLAY OF THE SETS YOU HAVE CREATED SO FAR. 



::ET count COMM-ND DESCRIPTION 



RELEVANCE 



1 


6 


SELECT 


2 


8 


SELECT . 


3 


14 ■■ 


SELECT 


4 


0 


. COMBINE 


5 


. ■ 22' 


COMBINE 


6 


^\ 


* COM':<INE 


7 


14 


COMBINE 



EMISSION FACTORS 
POLLUTANTS 
AEROSOLS 

1 ♦,2 

=EMISSIDN FACTORS 

2 3 

=PDLLUTRNTS ^ AEROSOLS 



^ 



POLLUTANTS 



=EMISSIDr^ FACTOR . + hEPDSOLS 
1 -t- 2 

=EMISrjION FACTOPr -k POLLUTANTS 
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* BRANCHING TQ DTHER PHASES IS NOW POSSIBLE..' 
ENTER ONE DF THESE UPTIDNS BY NUMBER: 

1. SATISFIEDt want TD PRINT SOME ITEMS TQ VIEW BY 
6QING- DN TD PHASE 3. 

S. NEED MORE DESCRIPTORS!^ WAf^T TO RETURN TO PHASE 1 • 
AND SELECT* E'APmTir OP RELATE MOPE TERMS. 

3. NEED MORE LIMITED SETS.* WANT TO RETURN TD PHASE £ 
TO COMBINE MORE SETS. 

•4". DISSATISFIED^ WANT TO START SAME SEARCH DVEP BY 
RETURNING TD BEGINNING. 

5. UNHAPPY* WANT TD START NEW SEARCH BY RETURNING 
TO THE BEGINNING. 

6. UNSURE, NEED HELP. 

YOU HAVE NDW ENTERED PHASE 3. — 

IN PHASE 3 J THE PRINT CDMMAND IS -AVAILABLE. ■ 

FROM THE SET HTSTDRY JUST PRINTED, CHDDSE A SET TO VIEW. 

ENTER A PRINT CDMMAM) FOR THAT SET. 
TPRINT 7/3/1-3 

ACCESS ND. = 53 

05=NONE 
06-JACKO R B 
06=NEUENDORFT D W 
06=BLftNDFDRD J R 

Q9=AIR l^OLLUTIDN CDNTRDL ASSOC ANNU MEET 69TH PORTLAND DREG 1976 

10=1976 ' ■ . X 

11 ♦PREPRINT 

16=EMISSIDN SOURCES 

13=LHB 

19=PARTICULATES 
19=YEAR 1976 
19=CDKE DVENS 
19=EMISSIDN FACTORS ' 
35^84765 

35^J«cKD Raitwvrr B., Dbvid W. NEueNDOPrr «nd Jomk R. BuftNEPn^D 
36^THE BY-PRODUCT COKE JVEN PUSHING DPERA^TIDN: TOTAL AND TRACE METAL. 
36*PAPTICULATE EMISSIONS. *J Axf» Pat-i-UT4taN Cdnti»du Assnc. » Pittsburgh, Pr 
36^15p.r 1976. <Pf»iescNTED «t the- Air Pai-t-UTiDH Chnttol. AssnciftTiON Anni 

JUDGE THE RELEVANCE DF THIS RECORD BY ENTERING 
AN INTEGER BETWEEN 1 AND 5 WHEREr 

1 = IRRELEVANT , 5 = MCfST RELEVANT 
?l ■ . 



1.6 . 
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ACCESS NO. = +0 * 

^^^□NTRftCT 
^slftiHR- M T 
06=LEE R . 
06=SCHUL2 E J ■ • 

10=1S^75 

}6=EMISSrDrf SOURCES 
ia=FLD 

l'?=PfiRTrCULftTES ^ * 

19=YEfiR 1975 * , 

19=PETRDLEUM REFIMIN6 
19=EMISSIDN FftCTDRS 
19=PftRTICLE SIZE DISTRXBUTIDN 
19=SULFUP DIOXIDE 
•35*84742 

35*'DftNftT Tew»Yr R. W. Lee» bno E. J. ScHuur f 
36*PflRTICULftTE fl«D SOa EMISSIONS FROM PROCESS HEATERS ftT SHELL AND T^Xfi 
36*REFINGRIES^ fWftCDRTES. UlftSHI MGTOrf . *t B^tteuue^ Pacific NaF^rHwesT -lSko 

JUDGE THE RELEVfmCE OF THIS RECORD BY Er4TERINe 
AN INTEGER BETWEEN 1 AND 5 WHERE: 

1 = IRRELEVANT r 5 - MOST RELEVANT 

•-•3 - " 



ACCESS 'NO. 32 

05=NDh"S 
06=WARD D E 
06=MCMftHON C K 

06= JDHANS EN R W > 

09=AIR POLLUTION CONTROL ASSOC ANNU MEET 69TH- POl^TLAND OREG 1976 

10=1976 

11 ♦PREPRINT 

16=EMISSI0N SOURCES 

13=THED 

i3=BIB 

19=P ARTICULATES 
19=F0PESTS 

49=YEAP 1976 • . . 

19=BURNIN6 

t9=EMISSIDN FACTORS • • 

35*64717 - - - 

35*WRf»B» Dftwu-D E.^ Chiwi.es K. McMftHOW? «nd Rn<3N«»» W. Jawf«>*SEN 
36*AN lj?DATE ON PARTICULATE EMISSIONS FROM FOREST FIRES. «t PREPRINT? THE 
36'»Aif» PoLuuTinN Cdnttol As-soc. ^ Pxtts^upghh Pt*. ISp.ir 1976. 20 hefs. 
36*- ^ Presented 'ftr the Air^ PnuLUTioN Cqntwdu fissuciRTinn ANNUr^ Meet-ins^ i 

JUDGE THE RELEVANCE- OF THIS RECORD BY E-NTERING 
AN INTEGER BETWEEN 1 AND 5 WHERE: 

1 = IRRELEVANT r 5 = MOST RELEVANT 

75 
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DQ YQU WISH- TO VIEW MDPE SETS? 
IF SQy ENTER flUDTHER- PRINT 'COMMAND. 
IF mJr ENTER /DONE. 
?PRINT 6y3yi 

ACCESS NO. = 53 

05=NONE 
06=JflCKQ R B 
06=NEUENDORFT D y 
06=BLflNriFDRD J R 

y^:^*.5g/°LLUTI0N CONTROL ^SSOC ANNU MEET 69TH PORTLAND ORES 1976 

11 ♦PREPRINT 

16=EMISSI0N SOURCES 
13=LflB 

19=/>ftRTICULft-TES 
19=YEflP 1976 
19=C0KE OVENS 
VI EMISSION FACTORS 
.35^84765 

lirJAlT?nTa?p^piTS?'Jn«S'^"/^'-^'^^^^ DPERftTION:- TOTAL AND TRACE METAL 
■StlSp . t^I^ EMISSIONS.,-. Ri^ PDi.L.UTiaN CnNXPot. Assoc., Pxtts,upgk, P, 

JUDGE THE RELEVANCE OF THIS RECORD BY ENTERING 
HN INTEGER BETWEEH 1 AND 5 WHERE-* ' 
1 = IRRELEVANT r 5 ^ MOST .RELEVANT 

?5 . " 

DO YOU WISH TO VIEW MORE SETS'? 
IF SO, ENTER ANOTHER PRINT COMMAND. 
IF NOTf ENTER /DONE. 
T'^DONE ' • '■ 

THE FOLLOWING CHOICES ARE NOW AVAILABLE. 

i' 'S2S?^ ^'"^^ DESCRIPTORS, EITHER FROM THE MATERIAL JUST VIEWED, OR Nl 

2. COMBINE K-SCRIPTDRS ALREADY CHOSEN IN NEW WAYS. ^^^W^D, OR Nl 

3. PRINT MDP*£ JiECORDS IF YOU HAVE NOT SEEN THEM f^iLL 

4. DECIDE YOU ARE SATISFIED AND YOUR SEARCH IS COMPLETE 

5. REQUEST HELP. ^u. ir-uc i c. 

MAKE YOUR CHOICE AND ENTER ITS NUMBER. 
?4 



^ IS:; 
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NDy THftT YOU CONSIDER YOUR SEftffCH- COMPLETEr 
WOULD YOU CHOOSE ONE OF THE FOLLOW IMG 
THftT MOST NEftPLY FITS YOUR EVftLUf^TIDN OF 
THE RECORDS RETRIEVED FOR YQU. 
OF THE RECORDS I Sfihtr I CONSIDER 
1; cESS THftN 10?: Ta HftVE BEEN RELEVANT. 
^2. 10-25?: WERE RELEVANT. 

3. 25-50?', WERE RELEVANT. 

4. 50-75?i WERE RELEVANT. 

5. 75-^0?: WERE RELEVANT, 

6. •50-100?'. WERE RELEVANT. 
?3 

SINCE YOU FEEL YOU DIDN'T FIND MANY RELEVANT ITErS 
DO YOU SUPPOSE THERE. IS NOTHINS IN THE LITERf^TUPE 
OR DP ^.'GJ THU1K ft REVIEW OF THE TERMS IN THE 
ftUTHUri .TY LIST WOULD BE HELPFUL? 
THESE QUESTIONS ARE FDR YOUR USE .lb THE FUTURE. 

THANK YQU? I HOPE YOU ENJOYED YOUR SEARCH. 



REVIEW OF COMMANDS AND SEARCH STA6ES 



;Tfii3E LINE COMMAND 



1 SELECT EMISSION FACTORS 

2 SELECT POLLUTANTS 

3 SELECT AEROSOLS 

4 EXPAND ' AEROSOLS 

5 RELATE AEROSOLS 

6 COMBINE 1*2 

7 COMBINE 2 3 

8 COMBINE 1 3 
9- COMBINE 1-1-2 

10 PRINT 7/3/1-3 

11 PRINT 6/3/1 



1 
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REVIEW OF SETS CREATED 



SET 


COUNT 


COMMfiMD 








1 


6 


SELECT 


2 


.8 


SELECT 


3 


14 


SELECT 


4 


0 


COMBINE 


5 


22 


COMBINE 


6 


20 


COMBINE 


7 


14 


COMBINE 



EMISSION FftCTDRS 
POLLUTRNTS 
flEPDSOLS 
1 2 

=EMISSION FACTORS POLLUTANTS 
2^-3 

=POLLUTfiNTS -h AEROSOLS 
t +■ 3 

^EMISSION F|^:TDRS +■ AEROSOLS 
1 2 

=EMrSSIDN FACTORS +■ POLLUTANT?^ 



PELB/flNCE 



REVIEW OF DESCRIPTORS USED 



DESCRIPTOR 
USED 



USED IN SEEN IN USED' IN 
SELECT EXPAND RELATE 



EK 7 SSI ON FACTORS 

POLLUTANTS 

AEROSOLS 



X 
,v 

X 



REVIEW OF ERRORS MADE 

FOLLOWS • TEXT OF 
LINE J? ERROR 

-5 DONE 
7 DONE /' 

HIT CARRIAGE RETURN t PLEASE ? 
TIME: 11.31 SECS. 



COMMAND 
ATTEMPTED 

PRINT 
COMBINE 



ERROR CODE 
<Tn BE TEXT) 

14 
1 



1.7. 
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